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PREFACE 


This  report  describes  a  study  of  the  implementation  of  innovative 
practices  in  Air  Force  depot  maintenance  contracts.  Through  its 
Contract  Repair  Enhancement  Program  (CREP),  today  known  as  the 
Contract  Repair  Process  (CRP),  the  U.S.  Air  Force  has  encouraged 
contracting  personnel  to  formulate  more  innovative  contracts  aimed 
at  improving  contractors’  performance,  particularly  regarding  cost 
and  schedule.  Successful  implementation  of  this  reform  requires  a 
transformation  in  organizational  structures  and  processes  because 
implementation  requires  individuals  to  change  fundamental  busi¬ 
ness  practices.  This  study  focused  on  the  relationship  between  or¬ 
ganizational  levers — those  aspects  of  the  work  environment  that  se¬ 
nior  leadership  could  affect  to  encourage  personnel  to  implement 
reform — and  contract  innovation.  These  organizational  levers  were 
measured  through  a  survey  of  contracting  personnel  at  Warner 
Robins  Air  Logistics  Center  (ALC).  Next,  the  study  team  designed  a 
set  of  regression  models  to  examine  whether  these  levers  were 
related  to  innovation  in  repair  contracts. 

The  research  reported  here  is  one  element  of  the  RAND 
Corporation's  ongoing  work  on  improved  contracting  sponsored  by 
the  Office  of  the  Secretary  of  the  Air  Force,  Deputy  Assistant 
Secretary  for  Contracting  (SAF/AQC).  The  study  was  conducted  in 
the  Resource  Management  Program  of  RAND  Project  AIR  FORCE. 

This  report  may  be  of  interest  to  those  concerned  with  acquisition 
reform  improvements  to  Air  Force  repair  contracts,  organizational 
transformation,  and  implementation  of  acquisition  reform  contract¬ 
ing  practices. 
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In  the  last  decade,  RAND  Project  AIR  FORCE  has  been  helping  the  Air 

Force  reshape  its  sourcing  policies  and  practices.  Readers  may  also 

be  interested  in  the  following  related  reports: 

•  Implementing  Best  Purchasing  and  Supply  Management 
Practices:  Lessons  from  Innovative  Commercial  Firms,  Nancy  Y. 
Moore,  Laura  H.  Baldwin,  Frank  Camm,  and  Cynthia  Cook, 
RAND,  DB-334-AF,  2002,  which  can  be  downloaded  from 
www.rand.org/publications/DB/DB334. 

•  Implementing  Performance-Based  Services  Acquisition  (PBSA): 
Perspectives  from  an  Air  Logistics  Center  and  a  Product  Center, 
John  Ausink,  Laura  H.  Baldwin,  Sarah  Hunter,  and  Chad  Shirley, 
RAND,  DB-388-AF,  2002,  which  can  be  downloaded  from 
www.rand.org/publications/DB/DB388. 

•  Federal  Contract  Bundling:  A  Framework  for  Making  and 
Justifying  Decisions  for  Purchased  Services,  Laura  H.  Baldwin, 
Frank  Camm,  and  Nancy  Y.  Moore,  RAND,  MR-1224-AF,  2001, 
which  can  be  downloaded  from  www.rand.org/publications/ 
MR/MR1224. 

•  Performance-Based  Contracting  in  the  Air  Force:  A  Report  on 
Experiences  in  the  Field,  fohn  Ausink,  Frank  Camm,  and  Charles 
Cannon,  RAND,  DB-342-AF,  2001,  which  can  be  downloaded 
from  www.rand.org/publications/DB/DB342 . 

•  Strategic  Sourcing:  Measuring  and  Managing  Performance,  Laura 
H.  Baldwin,  Frank  Camm,  and  Nancy  Y.  Moore,  RAND,  DB-287- 
AF,  2000,  which  can  be  downloaded  from  www.rand.org/ 
publications/DB/DB287. 

•  Incentives  to  Undertake  Sourcing  Studies  in  the  Air  Force,  Laura 
H.  Baldwin,  Frank  Camm,  Edward  Keating,  and  Ellen  M.  Pint, 
RAND,  DB-240-AF,  1998. 

•  Strategic  Sourcing:  Theory  and  Evidence  from  Economics  and 
Business  Management,  Ellen  M.  Pint  and  Laura  H.  Baldwin, 
RAND,  MR-865-AF,  1997. 
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RAND  Project  AIR  FORCE  (PAF),  a  division  of  the  RAND  Corporation, 
is  the  Air  Force  federally  funded  research  and  development  center  for 
studies  and  analyses.  PAF  provides  the  Air  Force  with  independent 
analyses  of  policy  alternatives  affecting  the  development, 
employment,  combat  readiness,  and  support  of  current  and  future 
aerospace  forces.  Research  is  performed  in  four  programs: 
Aerospace  Force  Development;  Manpower,  Personnel,  and  Training; 
Resource  Management;  and  Strategy  and  Doctrine. 

Additional  information  about  PAF  may  be  found  on  our  web  site  at 
http:  /  /  www.rand.org/paf. 
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SUMMARY 


In  1996,  the  Air  Force  adopted  the  Contract  Repair  Enhancement 
Program  (CREP),  today  known  as  the  Contract  Repair  Process  (CRP), 
which  consisted  of  a  series  of  contracting  reform  measures  intended 
to  respond  directly  to  customer  demands  at  the  same  time  as  reduc¬ 
ing  inventory,  process  steps,  lead  time,  and  total  system  operating 
costs.  Further,  the  Air  Force  aimed  to  accomplish  these  reform 
measures  while  maintaining  or  improving  readiness.  These  new 
practices  were  modeled  on  earlier  Air  Force  changes  in  acquisition 
practices  that  led  to  cost  improvements  and  accelerated  program 
schedules  for  acquiring  major  weapon  systems.  Concerned  by  what 
appeared  to  be  the  initially  slow  pace  at  which  depot  personnel  were 
carrying  out  the  CREP  reforms,  the  Air  Force  asked  RAND  to  assess 
what  might  be  hindering  their  implementation. 

PURPOSE  AND  APPROACH 

In  the  Air  Force,  as  in  the  private  sector,  senior  leadership  plays  a  vi¬ 
tal  role  in  instigating  change  in  business  practices.  To  do  so,  leaders 
generally  use  a  range  of  tools,  or  organizational  levers,  designed  to 
motivate  personnel.  Such  levers  might  include  setting  new  goals  and 
objectives,  communicating  them  throughout  all  levels  of  the  organi¬ 
zation,  and  changing  performance  evaluation  and  incentives  in  order 
to  encourage  their  adoption.  Particular  to  the  set  of  reform  measures 
that  the  Air  Force  planned  to  undertake  was  increased  teaming 
among  personnel.  Similarly,  success  at  the  operating  level,  which  is 
where  most  repair  contracts  are  written,  depends  on  how  effectively 
senior  leadership  can  move  personnel  toward  continued  use  of  new 
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practices  after  their  initial  introduction.  The  authors  analyzed 
whether  the  Air  Force’s  existing  organizational  levers  resulted  in  the 
use  of  CREP’s  innovative  reforms  within  depot-level  repair  contracts. 
We  focused  on  Warner  Robins  Air  Logistics  Center  (ALC)  because  it 
had  aggressively  pursued  the  incorporation  of  CREP  tenets  in  its  re¬ 
pair  contracts.  It  appeared  that  Warner  Robins  represented  the  ALC 
most  likely  to  have  the  largest  sample  size  and  most  variation  in  the 
number  of  CREP  tenets  incorporated  in  repair  contracts. 

Our  study  used  a  three-step  approach.  Step  one  consisted  of  a  litera¬ 
ture  review  on  organizational  levers  and  innovation  in  the  private 
sector,  combined  with  interviews  with  personnel  considered  to  have 
excelled  at  incorporating  CREP  tenets  in  the  first  CREP  contracts. 
The  literature  review  and  interviews  formed  the  basis  for  step  two,  a 
survey  of  key  personnel  participating  on  contract  repair  teams 
(CRTs),  i.e.,  program  managers,  procurement  contracting  officers, 
production  management  specialists,  and  item  management  special¬ 
ists.  Step  three  involved  regression  analyses.  The  survey  was  used  to 
measure  organizational  levers,  which  became  the  independent  vari¬ 
ables.  Survey  data  were  used  in  combination  with  reported  CREP 
tenets — the  dependent  variables — in  the  regression  analyses.  CREP 
tenets  were  classified  by  tenet  groups — simple  modifications,  key 
acquisition  reform  concepts,  complete  acquisition  reform  concepts, 
agile  logistics,  and  all  CREP  tenets — which  became  the  dependent 
variables.  Regression  analyses  were  carried  out  to  determine  the  re¬ 
lationship  between  each  organizational  lever  and  the  number  of 
CREP  tenets  incorporated  in  the  contract.  Each  step  is  discussed  in 
more  detail  below. 

Literature  Review  and  Interviews 

In  step  one,  we  reviewed  the  business  and  management  science  lit¬ 
erature  on  the  kinds  of  organizational  levers  senior  leadership  uses  to 
motivate  private-sector  enterprises  to  adopt  new  business  practices. 
The  literature  review  helped  to  identify  organizational  levers  to  be 
measured  in  the  personnel  survey.  In  turn,  the  review  helped  struc¬ 
ture  our  subsequent  interviews  with  CREP  contracting  teams  at 
Warner  Robins  ALC.  Similarly,  the  interviews  helped  us  understand 
how  these  levers  operated  within  the  ALC  and  the  CREP  initiative 
and  influenced  the  development  of  the  next  phase,  the  survey.  We 
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asked  CRT  personnel  to  talk  about  specific  CREP  contracts  and  asked 
them  questions  related  to  organizational  levers  within  these  exam¬ 
ples  of  contract  innovation.  The  interviews  helped  us  to  develop 
questions  using  language  and  context  we  believed  were  likely  to  res¬ 
onate  with  potential  survey  participants,  thus  preparing  us  for  step 
two. 

Survey  Development  and  Administration 

In  step  two,  based  on  inputs  from  the  literature  review  and  inter¬ 
views  conducted  at  Warner  Robins  ALC,  we  created  a  survey  instru¬ 
ment  designed  to  gather  information  on  organizational  levers.  The 
survey  asked  questions  in  five  categories  that  supplied  data  for  eight 
organizational  lever  variables.  Those  categories  were  (1)  attitude  to¬ 
ward  acquisition  reform,  (2)  leadership  support,  (3)  performance 
evaluation  and  rewards,  (4)  teaming  and  partnering,  and  (5)  training 
and  career  development.  We  surveyed  key  members  of  contract  re¬ 
pair  teams.  The  survey  was  conducted  first  in  1998  and  again  in  1999 
for  only  those  participants  who  had  missing  data  in  the  first  round  or 
failed  to  respond.  Each  participant  provided  only  one  set  of 
responses. 

Analyses 

In  step  three,  regression  analyses  showed  that  there  was  a  relation¬ 
ship  between  organizational  levers  and  reported  CREP  tenet  use. 
Between  December  1996  and  September  1998,  for  each  CREP  con¬ 
tract,  the  ALCs  considered  the  incorporation  of  16  tenets  identified 
by  HQ  Air  Force  Materiel  Command  (AFMC)  as  innovative  and  con¬ 
sistent  with  CREP  goals.  These  reported  tenets  were  used  as  the 
measure  of  contract  innovation.  The  research  process  involved  sur¬ 
vey  data,  i.e.,  organizational  levers  that  served  as  independent  vari¬ 
ables  in  the  regression.  The  process  also  involved  tenet  data,  which 
served  as  dependent  variables  in  the  regression  analyses. 

Step  3A.  Prior  to  the  regression  analyses,  Principal  Components 
Analysis  (PCA)  was  used  to  organize  responses  to  organizational  levers 
(the  independent  variables).  PCA  allows  the  analyst  to  examine  rela¬ 
tionships  among  item  responses  to  determine  whether  particular 
questions  reflect  the  same  underlying  concept.  The  survey  con- 
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tained  questions  on  organizational  lever  themes;  thus,  PCA  provided 
a  technique  to  group  questions  along  these  themes.  The  PCA  tech¬ 
nique  helped  identify  eight  organizational  levers  that  were  called: 

•  Attitude  toward  acquisition  reform 

•  Leadership  consistency 

•  Performance  evaluation 

•  Performance  incentives 

•  Effective  teaming 

•  Contractor  partnering 

•  Air  Force  partnering 

•  Training  in  acquisition  reform. 

Two  other  variables,  job  experience  and  a  U-2  dummy  variable  for 
the  U-2  Product  Directorate,  since  renamed  the  Intelligence, 
Surveillance,  and  Reconnaissance  Management  Directorate,  were 
computed  directly  and  did  not  require  PCA. 

Step  3B.  CREP  Tenet  Groups  (the  dependent  variables).  The  study 
used  CREP  tenets  as  an  element  of  the  dependent  variables,  which 
were  defined  by  AFMC  as  its  measure  of  contract  innovation  and 
collected  throughout  the  initiative.  We  received  a  record  of  the 
tenets  Warner  Robins  ALC  incorporated  in  their  CREP  contracts  from 
HQ  AFMC.  Individual  CREP  tenets  were  not  used  as  dependent 
variables,  because  they  lacked  policy  significance  by  themselves. 
The  question  became  one  of  how  to  group  CREP  tenets  in  ways  that 
would  have  strategic  policy  relevance  to  decisionmakers,  i.e.,  to  high- 
level  policy  goals  and  objectives.  We  analyzed  the  tenet  data  using 
cluster  analysis  to  discern  the  natural  groups  of  tenets,  but  the  clus¬ 
ter  analysis  showed  tenet  clusters  that  had  no  policy  relevance.  In 
the  end,  we  chose  to  use  as  dependent  variables  groups  of  CREP 
tenets  that  were  broadly  defined  by  the  CREP  initiative  itself  and  had 
strategic  policy  relevance,  plus  a  fifth  group  that  included  all  CREP 
tenets.  Some  tenets  were  included  in  more  than  one  group. 

The  first  innovation  group,  called  simple  modifications,  included  the 
easiest  tenets  to  implement  with  still-active  contracts,  usually  related 
to  speeding  up  transportation.  The  second  group,  called  key  acqui- 
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sition  reform  concepts,  consisted  of  reform  tenets  that  were  consis¬ 
tent  with  acquisition  reform  goals  and  could  be  measured  more  ob¬ 
jectively.  The  third  group,  called  complete  acquisition  reform  con¬ 
cepts,  measured  all  reform  tenets  included  in  the  second  group,  plus 
other  acquisition  reform-related  tenets  that  involved  subjective  mea¬ 
sures.  The  fourth  group,  called  agile  logistics,  included  all  tenets  de¬ 
signed  to  reduce  logistics  pipelines  by  speeding  up  the  repair  and 
transportation  pipeline  segments,  as  well  as  improving  other  logis¬ 
tics  efficiencies.  The  fifth  group,  called  all  tenets,  included  the  com¬ 
plete  set  of  CREP  tenets.  These  five  groups  of  tenets  became  the  de¬ 
pendent  variables. 

Step3C.  Regression  Analyses.  We  conducted  five  separate  regression 
analyses  to  determine  the  relationship  between  the  organizational 
levers  used  at  Warner  Robins  and  the  incorporation  of  CREP  tenets 
in  repair  contracts,  designed  to  improve  total  weapon  system  costs 
and  readiness.  The  regression  analyses  showed  that  some  organiza¬ 
tional  levers  help  explain  the  incorporation  of  CREP  tenets  in  repair 
contracts.  Thus,  this  study  demonstrated  that  senior  leadership  can 
influence  contract  innovation  through  organizational  levers,  al¬ 
though  not  always  in  expected  ways.  (See  pp.  37-43. ) 

RESULTS:  ORGANIZATIONAL  LEVERS  RELATED  TO 
CONTRACT  INNOVATION 

Our  analyses  showed  that  organizational  levers  help  explain  the  de¬ 
gree  of  tenet  use  that  teams  achieve  with  repair  and  sustainment 
contracts.  The  tenets  ranged  from  easy-to-implement  transportation 
improvements  to  more  difficult  acquisition  reform  measures,  such  as 
early  contractor  involvement  in  the  contracting  process.  In  some  ar¬ 
eas,  the  levers  were  positively  related  to  the  use  of  CREP  tenets;  in 
others,  the  levers  were  negative;  and  in  a  few,  they  had  no  influence 
at  all.  The  findings  were  as  follows: 

•  Training  in  acquisition  reform  had  a  consistent  and  positive 
statistical  relationship  with  tenet  use.  The  statistical  analyses 
showed  significance  in  all  four  groups  of  CREP  tenets  along  with 
the  group  of  all  CREP  tenets.  CRT  personnel  who  receive  more 
training  used  more  tenets  in  their  contracts  compared  with 
personnel  with  less  training.  (See  p.  38.) 
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•  Attitude  toward  acquisition  reform  also  had  a  consistent, 
positive  statistical  relationship  on  reported  tenet  use  in  all  tenet 
groups,  except  as  it  related  to  key  acquisition  reform.  These 
results  suggest  that  contract  teams  that  view  reform  more 
positively  or  agree  with  the  goals  of  the  initiative  also  implement 
more  reform  tenets  in  their  contracts.  (See  p.  40. ) 

•  Effective  teaming  had  a  negative  statistical  relationship  with 
reported  tenet  use.  Contract  teams  that  reported  effective 
teaming  also  implemented  fewer  tenets.  (See  p.  38. ) 

•  Contractor  partnering  had  a  positive  statistical  relationship  with 
simple  modifications  and  agile  logistics  innovation  goals.  CRTs 
that  perceived  contractors  to  be  better  partners  included  more 
CREP  tenets  in  their  contracts.  This  result,  however,  did  not 
occur  with  either  of  the  two  acquisition  reform  tenet  models. 
(See  p.  40. ) 

•  Leadership  consistency  had  a  positive  statistical  relationship 
with  tenet  implementation.  CRTs  that  perceived  consistent 
messages  throughout  management  and  believed  reform  would 
be  around  for  some  time  incorporated  more  agile  logistics  tenets 
in  their  contracts.  (See  pp.  40-41. ) 

•  Performance  evaluation  had  a  surprisingly  negative  statistical 
relationship  on  CRT  behavior  with  respect  to  agile  logistics  tenets 
and  no  significant  relationship  elsewhere.  (See  p.  41. ) 

•  Job  experience  had  a  slight  negative  statistical  relationship  with 
the  simple  modifications  and  the  complete  acquisition  reform 
innovation  groups,  but  the  magnitude  was  so  minimal  as  to  have 
little  practical  consequence. 

Two  variables — performance  incentives  and  Air  Force  partnering — 
had  no  explanatory  power  in  our  analyses.  Perhaps  the  types  of  in¬ 
centives  we  included  in  the  survey  were  not  sufficiently  representa¬ 
tive  of  rewards  offered  or  perhaps  personnel  do  not  perceive  these 
rewards  as  effective.  The  lack  of  a  relationship  between  Air  Force 
partnering  and  the  incorporation  of  CREP  tenets,  however,  raises 
questions  about  the  Air  Force’s  view  of  itself  as  a  customer  and  its 
effect  on  innovation. 
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Finally,  we  also  tested  a  dummy  variable  that  took  into  account  dif¬ 
ferent  contracting  processes  and  contracts  that  occurred  in  the  U-2 
Product  Directorate  (now  the  Intelligence,  Surveillance,  and 
Reconnaissance,  or  ISR  Management  Directorate).  Specifically,  the 
U-2  Product  Directorate  produced  sustainment  contracts  that 
included  repair  services  rather  than  solely  repair  contracts.  Our 
analysis  found  that  the  fact  that  a  contract  was  written  at  the  U-2 
Product  Directorate  was  positively  associated  with  the  reported 
incorporation  of  complete  acquisition  reform  concepts  and  agile 
logistics  tenets  in  CREP  contracts.  At  the  time  of  this  study,  the  U-2 
Product  Directorate  reported  a  high  rate  of  tenet  use,  which  has  been 
substantiated  with  follow  up  discussions  at  Warner  Robins  ALC. 

LESSONS  FOR  THE  AIR  FORCE 

In  general,  the  results  suggest  that  the  Air  Force  should  continue  to 
make  effective  use  of  those  levers  associated  with  positive  results 
(especially  training  and  fostering  positive  attitudes  toward  acquisi¬ 
tion  reform) ,  while  revisiting  others  associated  with  negative  results 
(especially  performance  evaluation  and  teaming).  Specifically,  the 
Air  Force  could  do  more  to: 

•  Let  personnel  know  that  senior  leadership  wants  to  see  progress 
in  achieving  well- specified  contracting  goals 

•  Learn  how  to  create  effective  teams.  This  process  will  require 
training  in  group  problem-solving  and  working  with  others  from 
different  functional  backgrounds,  in  addition  to  educating  teams 
on  legal  and  policy  changes 

•  Align  personal  evaluation  criteria  with  reform  goals. 


EXTENDING  THESE  RESULTS  TO  OTHER  ALCS  AND 
INITIATIVES 

To  the  degree  that  the  CREP  initiative  is  representative  of  contract 
reform  efforts  in  general,  the  Air  Force  should  consider  reinforcing 
the  organizational  levers  that  influence  innovation  and  looking  more 
closely  at  those  that  seem  to  have  no  relationship  or  have  a  negative 
relationship  with  innovation.  (See  pp.  42-43. ) 
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Are  the  lessons  from  this  study  applicable  to  new  contract-related 
initiatives  or  to  other  Air  Logistics  Centers?  We  hypothesize  that  it  is 
reasonable  to  think  the  dynamics  between  organizational  levers  and 
contract  innovation  at  Warner  Robins  ALC  are  similar  to  the  dynam¬ 
ics  of  these  variables  at  both  Oklahoma  City  and  Ogden  ALCs. 
However,  it  is  also  possible  that,  because  this  study  analyzed  con¬ 
tracts  at  a  center  viewed  by  HQ  AFMC  as  particularly  innovative, 
these  results  would  not  apply  to  the  other  two  centers.  Also,  senior 
leadership  at  the  other  ALCs  may  have  had  a  different  impact  on  per¬ 
sonnel  behavior  at  their  particular  center.  One  would  have  to  con¬ 
duct  a  similar  analysis  at  the  other  ALCs  to  know  whether  Warner 
Robins  represents  a  unique  case. 

Could  the  same  relationship  between  levers  and  innovation  that  we 
find  under  the  CREP  initiative  occur  with  other  contract  initiatives? 
After  all,  the  Air  Force  has  taken  on  many  other  contracting-related 
initiatives  since  CREP.  Many  of  those  attempt  more  significant 
change  in  behaviors  and  even  organizational  structure,  such  as  the 
purchasing  and  supply  chain  management  (PSCM)  initiative.1  If 
senior  leadership  does  not  address  those  levers  identified  as  having 
no  relationship  or  having  a  negative  relationship  with  innovation, 
then  leadership  may  have  to  work  as  diligently — or  more,  depending 
on  the  initiative — to  implement  other  innovative  practices,  much  as 
it  did  for  CREP.  (See  pp.  43-44.) 

As  senior  leadership  considers  the  behavioral  implications  of  con¬ 
tracting  initiatives  such  as  CREP  in  the  future,  this  study  should 
provide  suggestions  as  to  what  leadership  needs  to  strengthen  or  un¬ 
derstand  better  if  it  wants  personnel  to  implement  contract  innova- 


Whe  PSCM  initiative,  demonstrated  on  the  F100  engine  at  Oklahoma  City  ALC  in 
FY2002,  has  as  its  objective  a  strategic  means  of  selecting  and  managing  suppliers  to 
provide  more  effective  and  efficient  support  to  the  warfighter.  While  consistent  with 
CREP  outcome  goals,  PSCM  tries  to  do  much  more  than  CREP.  PSCM  would  work 
with  purchasing  and  supply  chain  activities  from  an  enterprise-  to  operating-level  per¬ 
spective.  It  is  strategic  in  scope  and  implies  new  processes,  practices,  and  organiza¬ 
tional  structure.  Behavioral  implications  for  PSCM  are  more  significant  than  CREP. 
See  Talking  Paper  on  Adopting  Improved  Purchasing  and  Supply  Chain  Management, 
Headquarters  U.S.  Air  Force,  Installations  and  Logistics,  Supply  Chain  Integration  and 
Logistics  Transformation  (HAF/IL-I),  November  25,  2002.  Also,  see  Nancy  Y.  Moore  et 
al.,  Implementing  Purchasing  and  Supply  Management  Practices:  Lessons  from  Inno¬ 
vative  Commercial  Firms,  RAND,  DB-334-AF,  2002.  Available  at  http:/  /www.rand.org/ 
publications/DB/DB334/. 
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tion.  This  study  also  provides  an  analytical  approach  to  updating  the 
relationship  between  organizational  levers  and  new  forms  of  con¬ 
tract  innovation.  Since  this  study  began,  the  Air  Force  has  added 
contract-related  initiatives  that  are  more  complicated  than  CREP  in 
their  approach  and  expected  outcomes — for  example,  PSCM,  corpo¬ 
rate  contracting,  and  performance-based  contracting.  This  study 
thus  offers  a  methodology  that  can  analyze  the  relationship  between 
individual  organizational  levers  and  contract  innovation.  Organiza¬ 
tional  levers  and  their  relationship  with  behavior  and  innovation 
continue  to  be  of  great  interest  to  the  Air  Force,  especially  as  it 
transforms  to  the  changing  threat  environment  and  takes  on  ever 
more  aggressive  contract  innovation  and  PSCM  implementation 
efforts. 
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Chapter  One 

INTRODUCTION 


The  U.S.  Air  Force  has  tried  for  some  years  to  reform  business  and 
contracting  practices  by  attempting  to  implement  best  commercial 
practices  in  its  logistics  support  services.  Many  of  the  ideas  for  im¬ 
proving  the  way  it  purchases  these  products  and  services  come  un¬ 
der  the  heading  of  acquisition  reform  (AR)  or  more  recently,  acquisi¬ 
tion  excellence,  which  falls  under  the  Deputy  Assistant  Secretary  of 
the  Air  Force  (Contracting)  (SAF/AQC).  Acquisition  reform  showed 
that  best  practices  could  lead  to  dramatic  cost  improvements  and 
accelerated  program  schedules  in  the  acquisition  of  major  weapon 
systems,  such  as  aircraft  and  missiles.  Soon  after,  the  Air  Force  tried 
to  apply  similar  best  practices  to  depot-level  services  to  deliver  prod¬ 
ucts  and  services  that  were  “better,  faster,  and  cheaper.”  Specifically, 
it  wanted  to  bring  improved  savings  and  effectiveness  to  an  area  that 
has  proven  unusually  difficult  to  shrink  in  size — depot-level  mainte¬ 
nance  and  repair — while  still  meeting  readiness  goals.  However,  the 
pace  of  implementing  its  acquisition  reform  efforts  occurred  slowly. 
In  response,  the  Air  Force  asked  RAND  for  help  in  understanding 
what  might  be  hindering  personnel  from  implementing  reform  more 
quickly.  If  the  right  policies  were  in  place  but  the  pace  of  improve¬ 
ments  in  logistics  support  services  was  slower  than  desired,  then  a 
natural  place  to  look  for  explanation  is  in  those  tools  senior  leader¬ 
ship  use  to  foster  specific  personnel  behavior  and  ultimately  innova¬ 
tion.  In  the  business  world,  these  tools  are  frequently  termed 
“organizational  levers.”  Our  study  thus  looked  closely  at  the  organi¬ 
zational  levers  in  place  during  the  Air  Force’s  reform  efforts  and  their 
relationship  to  repair  contract  innovation. 


l 


2  Organizational  Levers  Can  Affect  Acquisition  Reform  Implementation 


BACKGROUND 

In  1995,  the  Air  Force  initiated  efforts  to  improve  logistics  through  its 
“lean  logistics”  model.1  Lean  logistics  was  meant  to  improve  aircraft 
availability  through  faster  repair  cycle  times  for  depot-level  re- 
pairables  (DLRs),  prioritized  repair,  and  faster  transportation.  Such 
improvements  might  also  reduce  the  size  and  cost  of  supply  inven¬ 
tories  and  thus  reduce  the  in-place  resources  and  assets  necessary  to 
support  deployed  forces,  a  signature  goal  of  the  efforts  that  followed 
and  eventually  incorporated  lean  logistics,  today  known  as  agile 
logistics.  Agile  logistics  better  captured  the  objective  of  “[restructur¬ 
ing]  the  worldwide  logistics  system  to  equip  operational 
commanders  and  their  combat  forces  with  increased  deployment 
speed,  range  and  maneuverability.”2  Over  the  next  year,  the  Air 
Force  Materiel  Command  (AFMC)  initiated  the  Depot  Repair 
Enhancement  Program  (DREP)  and  Aircraft  Repair  Enhancement 
Program  (AREP).  DREP’s  attempts  to  enhance  user-oriented  metrics, 
such  as  “not  mission-capable  due  to  supply”  and  AWP  (awaiting 
parts)  time,  and  initial  improvements  seen  in  the  performance  of  Air 
Force-owned,  organic  (noncontracted)  repair  activities  made  it  obvi¬ 
ous  that  aircraft  availability  objectives  could  not  be  met  without  also 
improving  contract  repair  processes.  In  fact,  the  long  contract-repair 
schedules  for  some  units  stymied  efforts  at  reducing  organic  repair 
schedules  for  other  units.  As  a  companion  to  these  programs,  in 
1995  AFMC  developed  the  Contract  Repair  Enhancement  Program 
(CREP)  to  mirror  the  agile  logistics  goals  at  the  contract-repair  level. 

In  1996,  independent  of  CREP,  the  Department  of  Defense  (DoD)  be¬ 
gan  AR  efforts.  Specifically,  DoD  extended  AR  goals  and  objectives 
that  had  been  developed  for  the  acquisition  of  large  weapon  systems 
to  the  purchasing  of  logistics  support.  In  comparison  with  lean  and 
agile  logistics  efforts,  which  were  originally  focused  on  improving  or¬ 
ganic  rather  than  contract  operations,  AR  paid  more  attention  to 
specific  contract  design  goals  and  processes  used  successfully  by 


lrrhe  PACER  LEAN  initiative  launched  in  1995  applied  lean  logistics  concepts  to 
selected  systems  at  each  Air  Logistics  Center  (ALC).  These  prototypes  were  supposed 
to  identify  process  changes  needed  to  speed  component  repair. 

p 

Lt  Gen  William  P.  Hallin,  “Agile  logistics:  Where  we’ve  been,  where  we’re  going,”  Air 
Force  News,  April  28,  1998. 
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private- sector  firms.  AR  emphasized  greater  contractor  participation 
in  specifying  the  repair  process,  the  elimination  of  unnecessary  data- 
reporting  requirements,  early  involvement  of  the  supplier  in  contract 
design  formulation,  and  other  practices  used  in  the  commercial 
world  to  reduce  costs  and  enhance  support. 

Because  of  the  similarity  in  goals  between  agile  logistics  and  acquisi¬ 
tion  reform,  acquisition  reform  tenets  were  added  to  CREP’s  already 
existing  agile  logistics  tenets  in  1996.  CREP  was  thus  designed  to  in¬ 
corporate  the  best  of  both  AR  and  lean  logistics,  adopting  the  faster 
repair  cycle  time  emphasis  of  lean  logistics  and  the  better  contract 
design  and  contract  process  objectives  of  acquisition  reform. 

When  the  data  for  this  study  were  collected  in  1998  and  1999,  CREP 
operated  under  the  assumption  that  major  changes  in  how  contracts 
were  written  should  improve  cycle  time,  cost,  and  quality.  To  en¬ 
courage  Contract  Repair  Teams  (CRTs)  to  consider  a  wide  spectrum 
of  innovations  that  were  consistent  with  acquisition  reform  and  agile 
logistics  goals,  HQ  AFMC  developed  a  checklist  of  16  CREP  tenets 
that  it  wanted  the  ALCs  to  consider  incorporating  in  their  CREP  con¬ 
tracts.  These  tenets  were  based  on  the  Air  Force’s  original  acquisi¬ 
tion  reform  initiatives  and  on  lean  logistics  reforms  (later  called  agile 
logistics  reforms). 

All  CREP  tenets  on  the  checklist  applied  to  at  least  one  of  the  im¬ 
provement  areas  mentioned  above,  i.e.,  improved  cycle  time,  cost, 
and  quality.  For  example,  CREP  called  for  functional  expert  teaming 
at  the  outset  and  early  contractor  involvement.  This  teaming  was  ul¬ 
timately  supposed  to  lead  to  better  coordination  between  require¬ 
ments  and  contract  conditions,  shorter  administrative  lead-time, 
and  improved  contracts,  along  with  a  reduction  in  overall  costs.  In 
addition,  as  part  of  its  AR  philosophy,  CREP  allowed  the  contractor 
more  say  in  how  repairs  were  accomplished.  CREP  also  removed 
non-value-added  and  unnecessary  government  requirements  in 
contracts  to  reduce  contractor  costs;  in  special  cases,  this  reform 
may  lead  to  more  competition  for  governmental  contracts  and  the 
entrance  of  new  suppliers.3  Many  of  the  overarching  objectives  of 


o 

Most  repair  contracts,  measured  in  dollars,  are  sole  source.  (Sole  sources  of  repair 
can  arise  if  the  contractor  owns  the  technical  data,  which  raises  a  barrier  to  new 
entrants,  or  if  the  market  is  limited,  such  as  with  obsolete  technology  or  items  with  low 
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CREP  were  embodied  in  the  use  of  innovative  contract  tenets  that 
were  reported  to  HQ  AFMC  by  the  CRTs.  The  tenets  were  a  practical 
means  of  translating  the  strategic  goals  of  “faster,  better,  cheaper” 
into  tactical  goals  that  could  be  measured  and  incorporated  in  con¬ 
tracts  and  the  contracting  process. 

One  goal  of  CREP — to  speed  repairs — has  the  potential  benefit  of 
cutting  overall  costs,  provided  the  increased  speed  costs  less  than 
replenishing  the  inventory  with  new  spares.  AFMC  developed  a  cost- 
benefit  analysis  tool  to  help  personnel  compare  the  cost  of  shorten¬ 
ing  repair  cycle  time  with  the  cost  of  new  inventory  associated  with 
the  current  repair  cycle  time.4  In  sum,  a  number  of  CREP  innova¬ 
tions  are  meant  to  ensure  the  Air  Force  provides  responsive  service 
while  improving  the  contract  itself  and  streamlining  the  contract 
process. 

Still,  as  noted  earlier,  reform  has  been  slow.  Thus,  to  identify  the 
barriers  to  acquisition  reform  implementation,  this  study  examined 
the  incorporation  of  CREP  tenets  in  repair  contracts  at  an  Air  Force 
Air  Logistics  Center.  The  study  tried  to  link  this  incorporation  to 
various  organizational  levers,  or  policies  and  practices  senior  leader¬ 
ship  has  at  its  disposal  to  encourage  and  influence  individual  behav¬ 
ior,  especially  behavior  at  the  operating  level  where  most  repair  con¬ 
tracts  are  written. 

At  present,  the  Air  Force  conducts  its  logistics  services  at  three  AFMC 
ALCs:  Oklahoma  City  in  Oklahoma,  Ogden  in  Utah,  and  Warner 
Robins  in  Georgia.  These  ALCs  provide  serviceable  parts  to  their  cus¬ 
tomers  through  the  purchase  of  new  parts  and  the  repair  of  unser¬ 
viceable  or  recoverable  items.  For  our  study,  we  chose  to  focus  on 
acquisition  reform  and  agile  logistics  implementation  at  Warner 
Robins  ALC  because  its  senior  leadership  had  aggressively  pursued 


failure  rates.)  Thus,  this  change  may  have  an  effect,  albeit  a  minor  one.  Primarily 
commercial  systems,  such  as  business  jets  used  by  senior  leadership  in  the  field, 
benefit  the  most  from  the  removal  of  military  specifications  and  standards.  Removal 
of  standards  and  specifications  on  combat-essential  weapon  systems  must  be  done 
carefully;  the  benefit  here  primarily  reduces  costs  to  the  sole-source  provider. 

4At  the  time  of  this  study,  only  about  10  percent  of  CREP  items  were  in  a  buy  position, 
which  meant  that  most  items  required  cycle-time  reductions  at  no  additional  cost.  In 
some  cases,  CRTs  accomplished  this  through  reengineering  the  repair  process,  which 
required  ALC  approval. 
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CREP  and  gave  it  significant  attention  from  the  beginning  of  the  ini¬ 
tiative.  Warner  Robins  also  officially  reported  the  second  highest 
number  of  CREP  contracts  to  HQ  AFMC,  providing  a  large  number  of 
observations  from  a  single  center.5 

Between  December  1996  and  September  1998,  each  ALC  evaluated 
its  repair  contracts  for  consistency  with  CREP  goals.6  The  evaluation 
consisted  of  determining  the  number  of  CREP  tenets  incorporated 
using  the  16-item  checklist  of  tenets  developed  from  the  ALCs’  initial 
experience  in  applying  agile  logistics  and  best  practices  to  their  first 
prototype  CREP  contracts.  After  the  establishment  of  the  checklist, 
HQ  AFMC  required  the  ALCs  to  report  quarterly  on  the  number  of 
these  tenets  they  had  incorporated  in  DLR  contracts,  either  through 
modifications  or  new  contracts  (more  details  about  16  goals  are  in 
Chapter  Two  of  this  report).  Although  the  metric  served  as  an 
administrative  device  to  track  the  pace  of  acquisition  reform  in 
repair  contracts,  its  criteria  for  classifying  the  inclusion  of  specific 
tenets  in  a  contract  were  not  clearly  defined,  as  reported  by  inter¬ 
viewed  Warner  Robins  ALC  personnel.7  CREP  tenet  reporting  ended 
in  September  1998.  This  metric  was  the  best  available  formal  Air 
Force  summary  measure  on  contracts  the  ALCs  consider  “improved” 
under  the  tenets  incorporated  on  those  contracts. 

NEEDED  BEHAVIORAL  CHANGES 

Improvements  in  the  contract  repair  processes  do  not  come  without 
some  tradeoffs  to  the  Air  Force.  There  are  aspects  of  CREP  that  may 
make  repair  more  costly  initially  or  locally.  On  the  other  hand,  these 


5In  September  1998,  the  five  Air  Force  ALCs  reported  the  following  total  number  of 
contracts  that  had  incorporated  CREP  tenets:  Ogden  ALC,  20;  Oklahoma  City  ALC,  54; 
Sacramento  ALC,  82;  San  Antonio  ALC,  229;  Warner  Robins  ALC,  101.  San  Antonio 
stands  out  because  it  appeared  to  report  CREP  tenets  on  contract  modifications  in 
addition  to  basic  contract  numbers,  unlike  the  other  four  ALCs.  If  one  were  restricted 
to  contract  numbers  alone,  San  Antonio’s  number  would  have  dropped  from  229  to 
152.  Viewed  this  way,  Warner  Robins  looked  like  a  good  place  to  study  CREP 
contracts. 

^Contract  Repair  Enhancement  Program  (CREP)  Phase  III  Implementation:  A  Briefing, 
Warner  Robins  ALC,  February  19 , 1997. 

7As  an  example,  one  CREP  tenet  is  to  reduce  military  standards  and  specifications.  It 
is  unclear  whether  the  reduction  of  one  standard  or  specification  was  enough  for  a 
CRT  to  report  success  in  using  this  tenet. 
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initiatives  may  ultimately  reduce  long-term  total  costs.  For  example, 
CREP  requires  that  the  contractor  assume  some  risk  in  terms  of 
equipment  and  manpower  availability — so  that  they  may  perform 
efficient  repair  work  on  short  notice  (i.e.,  “just-in-time”  repair  on 
demand).  In  turn,  for  work  in  which  there  is  only  one  contractor 
working  on  items  that  require  expensive  repair,  CRTs  considered  al¬ 
lowing  contractors  to  order  long-lead-time  material  at  Air  Force  ex¬ 
pense  (i.e.,  allowing  the  contractor  to  lay  in  parts  needed  for  the  re¬ 
pair  before  the  repair  actually  occurred).  These  concepts  are  not 
new,  but  their  use  on  certain  repair  contracts  appears  to  have  ex¬ 
panded.  Additionally,  working  toward  shorter  repair  cycle  times  in¬ 
volves  a  tradeoff.  Contractors  may  be  able  to  provide  a  product  more 
quickly — the  stated  goal  of  agile  logistics — but  it  may  come  with 
increased  unit  repair  cost  to  the  Air  Force.  These  innovations,  how¬ 
ever,  are  still  preferable  when  they  lead  to  overall  reduced  total 
ownership  costs. 

Implicit  in  some  CREP  tenets,  such  as  early  contractor  involvement, 
is  a  change  in  how  and  when  the  contractor  is  brought  into  the  con¬ 
tracting  process.  Laying  in  long-lead-time  material,  for  instance,  re¬ 
quired  senior  leadership  approval  and  also  implied  changes  to  the 
process.  For  teams  to  achieve  the  incorporation  of  some  tenets, 
teams  had  to  go  beyond  traditional  methods  and  had  to  build  busi¬ 
ness  cases  to  meet  requirements  in  new  ways  that  were  consistent 
with  CREP.  These  differences  in  solving  contracting  problems  were 
not  specifically  recorded  but  were  implicit  in  many  of  the  CREP 
tenets.  The  challenge  to  senior  leadership  was  to  encourage  individ¬ 
uals  to  take  the  new  practices  and  policies  provided  by  acquisition 
reform  and  agile  logistics  and  implement  innovation,  even  though 
methods  for  achieving  innovation — the  incorporation  of  CREP 
tenets — had  to  be  discovered,  learned,  and  refined.  The  implemen¬ 
tation  of  CREP  required  CRTs  to  behave  differently.  Senior  leader¬ 
ship  had  instructed  CRTs  to  implement  CREP  and  had  various  tools 
to  encourage  changed  behavior  from  CRT  team  members,  which  we 
turn  to  next. 

This  study  took  the  approach  that  senior  leadership  has  available  or¬ 
ganizational  levers  or  methods  by  which  it  can  influence  individual 
behavior  at  the  operational  level  for  contracting.  It  did  not  address 
process  changes  brought  about  by  this  initiative.  AFMC  adopted  the 
Integrated  Product  Team  (IPT)  approach  of  CREP  with  its  Contract 
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Repair  Teams,  along  with  the  early  acquisition  reform  and  agile  lo¬ 
gistics  goals  and  approaches  as  standard  practice.8  Some  of  the  most 
important  process  changes,  such  as  the  use  of  the  cost-benefit  analy¬ 
sis  tool  and  teaming,  were  available  to  all  CRTs  at  the  same  time.  Our 
study  focused  on  differences  in  CREP  tenet  incorporation  outcomes 
with  respect  to  teams  and  organizational  levers.  We  were 
particularly  interested  in  what  influenced  depot-level  personnel  to 
implement  CREP  and  what  leadership  could  do  to  further  encourage 
innovation  implementation.9 

This  report  considers  how  organizational  levers  are  associated  with 
change  in  contract  design  and  the  contract  process  through  their  re¬ 
lationship  with  personnel  attitudes  and  behaviors.  For  any  major 
change  in  business  practice  to  succeed  over  the  long  run,  preferred 
behavior  must  not  be  a  temporary  shift  or  allowed  to  revert  to  the  old 
ways  once  senior  leadership  attention  moves  on  to  another  initiative. 

In  a  separate  analysis  we  conducted  on  contracts  written  during  the 
CREP  initiative,  we  found  that  contracts  incorporating  CREP  tenets 
had  different  outcomes  than  non-CREP  contracts.  Data  on  recover¬ 
able  items  on  CREP  and  non-CREP  contracts  written  over  a  similar 
period  showed  statistically  unchanged  repair  prices  for  CREP  items 
compared  with  non-CREP  items.  On  average,  prices  increased  for 
non-CREP  items,  but  they  held  steady  for  CREP  components.  Nego¬ 
tiated  flow  days — the  repair  cycle  time  required  by  the  contract — for 
items  on  CREP  and  non-CREP  contracts  showed  real  improvements, 
i.e.,  they  were  both  less  than  their  respective  previous  contract  flow 
days.  Although  the  flow  days  for  non-CREP  items  improved  more 
than  for  CREP  items,  i.e.,  they  decreased  from  previous  flow  days, 


8Before  CREP,  ALC  personnel  processed  the  requirements,  writing  and  negotiating  the 
contract  in  a  series  of  sequential  steps.  Personnel  formally  assembled  jointly  at  the 
end  of  this  process  at  a  Contract  Management  Review  Board  (CMRB)  meeting.  With 
CREP,  personnel  met  as  a  Contract  Repair  Team  at  the  beginning  of  the  process  to 
make  decisions  collectively.  This  procedure  allowed  teams  to  make  major  changes  in 
contracts  that  otherwise  might  not  have  occurred  because  of  the  approval  needed  by 
all  functional  experts. 

9Factors  other  than  organizational  levers  may  have  accounted  for  the  incorporation  of 
CREP  tenets.  Technically,  these  other  factors  are  captured  in  the  regression  residual 
variables.  Recognizing  that  factors  other  than  organizational  levers  can  explain 
reform,  we  examined  unit  price  and  repair  flow  day  differences  in  CREP  and  non- 
CREP  contracts  written  about  the  same  time  (see  Appendix  A).  The  comparison 
captures  the  effect  of  all  factors  influencing  CRT  behavior  during  the  study  period. 


8  Organizational  Levers  Can  Affect  Acquisition  Reform  Implementation 


they  did  so  with  higher  prices.  Flow  days  for  CREP  items  decreased 
at  the  same  time  their  prices  remained  essentially  unchanged.  Ap¬ 
pendix  A  discusses  these  results. 

Even  though  the  introductory  phase  of  the  reform  process  is  over, 
opportunities  for  repair  contract  improvements  continue.10  As 
contracts  written  early  in  the  initiative  expire,  CRTs  have  a  chance  to 
apply  the  lessons  learned  to  date  to  write  even  better  contracts.  In 
addition,  as  best  practices  evolve  and  private-sector  firms  continu¬ 
ously  improve  costs,  product  quality,  and  schedule,  CRTs  can  benefit 
from  these  greater  efficiencies  as  successive  generations  of  CREP 
contracts  expire. 

As  senior  leadership  attention  moves  on  to  new  problems  and  issues, 
the  continued  pace  of  reform  implementation  will  depend  on  the 
extent  to  which  the  Air  Force  leadership  and  the  organization  can 
encourage  new,  desired  behavior.  This  study’s  findings  suggest  two 
types  of  lessons  from  CREP.  First,  our  results  suggest  that  at  Warner 
Robins  ALC,  many  of  the  senior  leadership’s  organizational  levers 
had  a  positive  relationship  with  contract  innovation.  These  levers 
are  a  sign  of  success  and  suggest  that  those  policies  and  practices 
AFMC  and  Warner  Robins  ALC  used  for  CREP  were  constructive. 
Second,  there  are  indications  that  other  organizational  levers  were 
not  related  to  innovation.  These  levers  need  to  be  investigated;  if 
they  are  not  understood  and  addressed,  they  could  continue  to  have 
a  negative  or  neutral  relationship  to  contract  innovation. 

Having  thus  outlined  the  CREP  initiative  that  served  as  the  focus  of 
our  study,  let  us  now  turn  to  the  next  chapter,  which  describes  this 
study’s  approach  and,  in  particular,  the  survey  conducted  at  Warner 
Robins  ALC  to  measure  organizational  levers.  Chapter  Three  pre¬ 
sents  the  results  of  the  regression  analyses  we  conducted  on  the  sur¬ 
vey  data  with  respect  to  tenet  use  and  discusses  the  significance  and 
implications  for  the  implementation  of  acquisition  reform  and  other 
Air  Force  initiatives.  A  series  of  appendices  provides  detailed  dis¬ 
cussions  on  technical  topics.  Appendix  A  shows  several  histograms 
describing  changes  in  price  and  schedule  for  items  in  CREP  and  non- 


10Even  though  the  Air  Force  has  changed  its  contract  repair  process  along  the  lines  of 
the  CREP  initiative,  CREP  has  been  a  clearinghouse  for  contract  reform  and  contract 
process  changes. 
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CREP  contracts.  Appendix  B  provides  a  copy  of  the  questions  used  to 
structure  interviews  conducted  in  April  1998  and  June  1998  with 
CRTs.  Appendix  C  provides  a  copy  of  the  web-based  survey  as  it 
appeared  to  participants.  Appendix  D  discusses  the  principal 
components  analyses  conducted  on  the  survey  data  used  to  create 
the  independent  variables — organizational  levers — in  the  regression 
analyses.  Finally,  Appendix  E  provides  the  detailed  results  of  the  re¬ 
gression  analyses. 


Chapter  Two 

METHODOLOGICAL  APPROACH 


Senior  leadership  can  play  a  vital  role  in  affecting  change  in  business 
practices  through  its  capacity  to  set  new  goals  and  objectives,  com¬ 
municate  them  throughout  all  levels  of  the  organization,  and  modify 
performance  evaluation  and  incentives  in  ways  that  encourage  per¬ 
sonnel  to  adopt  those  new  business  practices.  Success  at  the  operat¬ 
ing  level  depends  on  how  effectively  senior  leadership  can  move  per¬ 
sonnel  toward  changed  behavior  and  maintain  that  behavior  long 
after  an  initiative  becomes  standard  operating  policy.  The  tools  lead¬ 
ership  has  to  motivate  line-operator  personnel  are  what  this  study 
refers  to  as  “organizational  levers,”  i.e.,  those  means  by  which  lead¬ 
ership  encourages  and  influences  individual  behavior  to  achieve 
certain  policy  goals  and  objectives. 

This  chapter  describes  our  study  methodology;  in  particular,  we  de¬ 
scribe  our  process  of  measuring  the  organizational  levers  in  use  at 
Warner  Robins  ALC — levers  designed  to  stimulate  the  use  of  CREP 
tenets  in  repair  contracts.  We  then  outline  the  survey  we  conducted 
at  Warner  Robins  ALC  to  assess  behaviors  and  attitudes  toward  ac¬ 
quisition  reform.  Specifically,  the  survey  measured  organizational 
levers  at  Warner  Robins  ALC  during  the  CREP  initiative.  This  chapter 
also  describes  the  regression  models  we  developed  to  explore  the  re¬ 
lationship  between  these  levers  and  number  of  select  CREP  tenets 
incorporated  in  a  contract. 

Achieving  improvements  in  ALC  contract  repair  outcomes  is  com¬ 
plex.  The  large  variety  of  contracts  and  the  number  of  contractors 
with  whom  ALCs  work  mean  operating-level  personnel  must  have 
significant  motivation  to  translate  senior  leadership  policy  guidance 
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and  directives  into  each  repair  contract.  Improved  repair  outcomes 
from  contractors  do  not  happen  by  Air  Force  fiat,  particularly  be¬ 
cause  personnel  at  contract  repair  facilities  answer  to  their  own  cor¬ 
porate  leadership,  not  to  the  Air  Force.  Thus,  to  generate  better  out¬ 
comes,  Air  Force  senior  leadership  faces  challenges  to  find  ways  to 
motivate  its  contract  repair  teams  to  write  the  kinds  of  contracts  that 
ultimately  lead  to  improved  performance  from  contractors  and  to 
manage  the  Air  Force-contractor  relationship  from  a  mutually  bene¬ 
ficial  perspective. 

How  does  the  Air  Force  seek  to  galvanize  its  contract  repair  teams 
so  that  they  write  innovative  contracts  and  manage  the  Air  Force- 
contractor  relationship  more  effectively  and  consistently?  Senior 
leadership  generally  uses  a  series  of  levers  that  influence  specific 
kinds  of  organizational  practices  and  behavior.  If  these  orga¬ 
nizational  levers  are  aligned  and  operating  as  Air  Force  senior 
leadership  intends,  one  would  expect  to  see  the  kind  of  relationship 
between  levers  and  contract  outcomes  the  Air  Force  wants — i.e., 
organizational  levers  help  explain  contract  measures  of  success  with 
respect  to  the  contract  initiative,  such  as  CREP. 

To  examine  Warner  Robins  ALC’s  use  of  organizational  levers  and 
the  outcome  on  contract  innovation,  this  study  followed  three  steps. 
First,  we  conducted  a  literature  review  and  interviews  with  key  per¬ 
sonnel  at  an  ALC.  Second,  we  developed  a  survey  based  on  the  re¬ 
view  and  interviews  and  administered  it  to  personnel  on  CREP 
Contract  Repair  Teams.  Third,  we  conducted  a  series  of  analyses  that 
ultimately  established  through  empirical  means  a  relationship  be¬ 
tween  organizational  levers  and  a  contract  measure  of  innovation — 
that  is,  the  number  of  CREP  tenets  incorporated  in  repair  contracts. 
The  analyses  involved  a  three-part  process.  One  phase  of  the  analyt¬ 
ical  process  used  principal  components  analysis  (PCA)  on  the  survey 
responses  to  identify  questions  associated  with  specific  organiza¬ 
tional  levers  (the  independent  variables).  Another  phase  of  the  ana¬ 
lytical  process  developed  five  groups  of  specific  kinds  of  CREP  tenets 
incorporated  in  repair  contracts  (the  dependent  variables) .  The  final 
phase  of  the  analytical  process  involved  regression  analyses  to  de¬ 
termine  whether  organizational  levers  could  help  explain  the  extent 
to  which  these  grouped  tenets  were  incorporated  in  repair  contracts. 
These  CREP  tenets  were  considered  by  AFMC  as  improvements  to  be 
implemented  in  logistics  support  contracts  and  measures  of  innova- 
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tion.  The  reader  can  refer  to  Appendix  B  for  the  interview  questions 
and  Appendix  C  for  the  PCA  technique  used  to  determine  the  organi¬ 
zational  levers.  Each  of  these  steps  is  described  in  more  detail  below. 

LITERATURE  REVIEW  AND  INTERVIEWS 

The  Air  Force’s  shift  to  acquisition  reform  and  new  business  prac¬ 
tices  mirrors  private-sector  transformation  efforts  in  that  it  requires 
significant  changes  in  organizational  behaviors  and  attitudes.  The 
study  team  reviewed  the  literature  on  organizational  transformation 
to  identify  those  elements  considered  key  in  helping  commercial 
firms  transform  their  business  practices  and  organizations.  The  re¬ 
view  indicated  that  companies  that  have  successfully  pursued  large- 
scale  transformation  have  made  changes  to  all  of  their  major  organi¬ 
zational  subsystems,  including  work  structures  and  processes, 
training  and  rewards  systems,  and  organizational  culture.  The  CREP 
initiative  attempted  smaller-scale  changes  that  focused  on  encourag¬ 
ing  new  behaviors  from  the  organization  using  new  policies  and 
slightly  different  processes,  such  as  teaming,  cost-benefit  analyses, 
and  early  contractor  involvement. 

Literature  Review 

The  literature  on  successful  organizational  change  within  private- 
sector  firms  argues  that  a  key  element  to  instituting  such  transfor¬ 
mation  is  leadership  support  (Strebel,  1996;  Katzenbach  and  Smith, 
1993;  Larkin  and  Larkin,  1994).  All  levels  of  management  must 
openly  promote  a  consistent  platform  of  change  in  order  to  motivate 
employee  working  modes. 

Typically,  employees  resist  change  in  work  behavior  unless  incen¬ 
tives  are  in  place  to  reward  new  behaviors.  The  literature  suggests 
necessary  steps  for  incentives  to  change  behavior  successfully.  First, 
management  must  provide  clear  standards  of  evaluation  that  are 
aligned  with  performance  goals  (Pfeffer,  1996;  Strebel,  1996).  Next, 
Pfeffer  (1996)  and  Strebel  (1996)  suggest  that  leaders  must  provide 
feedback  on  whether  employee  performance  is  leading  to  better 
contract  performance.  Finally,  consistent  rewards  and/or  sanctions 
are  needed  to  reinforce  behavior  over  time  (Ghoshal  and  Bartlett, 
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1996).  The  effectiveness  of  rewards  will  be  realized  only  if  con¬ 
tracting  personnel  are  told  howto  achieve  them  (Pfeffer,  1996). 

To  ensure  that  new  contracting  practices  continue  to  be  imple¬ 
mented  over  time,  psychological  barriers  of  change  must  also  be  ad¬ 
dressed.  Part  of  leadership’s  responsibility  is  to  instill  the  under¬ 
standing  that  implementing  new  contracting  practices  will  lead  to 
improved  contract  performance.  Employees  need  “proof”  that  these 
new  methods  will  help  them  create  a  better  contract  (Larkin  and 
Larkin,  1994).  Addressing  these  needs  can  facilitate  the  continuation 
of  changed  behaviors  and  attitudes  past  the  introductory  phase. 

In  sum,  the  literature  on  successful  organizational  transformation 
suggests  that  permanent  changes  in  contracting  personnel  behavior 
will  come  only  through  consistent,  thorough  support  by  Air  Force 
leadership.  This  support  may  be  demonstrated  in  a  number  of  ways: 
consistent  communication  of  support  by  leadership,  incentives  for 
new  behaviors,  training  and  skill  development,  and  performance 
feedback.  These  factors  should  lead  to  changes  in  employee  atti¬ 
tudes  that  then  promote  innovation.  Changes  in  attitudes  will  help 
maintain  new  behaviors  over  time. 

A  review  of  the  literature  identified  at  the  highest,  general  level  the 
kinds  of  organizational  levers  that  private-sector  firms  have  found  to 
be  important  in  their  own  transformational  efforts.  The  levers  from 
the  review  became  topics  for  discussion  in  interviews  with  Warner 
Robins  ALC  personnel  who  were  selected  by  the  ALC  as  innovative  in 
writing  CREP  contracts.  Interviews  at  Warner  Robins  ALC  then 
helped  the  study  team  construct  a  survey  that  would  eventually  be 
used  to  measure  organizational  levers.  We  next  describe  the  inter¬ 
views. 

Interviews 

The  interviews  conducted  at  Warner  Robins  ALC  provided  the  study 
team  insight  on  how  organizational  levers  operated  at  the  center 
during  the  CREP  initiative.  We  were  interested  in  listening  to  the 
language  that  CRT  personnel  used  when  talking  about  these  levers 
and  the  issues  raised.  These  interviews  influenced  the  way  we 
framed  our  survey  questions. 
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The  study  team  conducted  three  sets  of  meetings  at  Warner  Robins 
ALC  to  accomplish  three  objectives.  During  the  first  set  of  meetings 
(March  10-11,  1998),  we  met  with  all  key  CREP  stakeholders  to  com¬ 
municate  our  study’s  goals  and  objectives  and  to  gain  product  direc¬ 
torate  leadership  approval  to  interview  their  personnel.  We  first  met 
with  personnel  at  Warner  Robins  ALC  who  could  help  us  gain  sup¬ 
port  of  the  product  directorate  chiefs  to  conduct  a  survey  of  their 
personnel.  In  particular,  we  met  with  the  chiefs  of  Plans  and 
Programs  (XP),  Contracts  (PK),  and  Reengineering  (RE).  (Warner 
Robins  ALC /RE  was  a  facilitating  office  for  processes  and  practices 
needed  to  implement  CREP.)  We  also  met  with  chiefs  and/or  their 
staff  in  the  then-product  directorates  of  Ground  Equipment  (LB),  C-5 
(LC),  F-15  (LF),  Electronic  Warfare  (LN),  U-2  (LR),  Special  Operations 
Forces  (LU),  and  Avionics  (LY).  In  these  meetings,  we  conveyed  the 
study’s  goals  and  established  contact  with  those  CRTs  considered  by 
Warner  Robins  ALC/PK  and  Warner  Robins  ALC/RE  as  especially  in¬ 
novative  in  their  CREP  implementation.  These  high-level  meetings 
were  critical  in  helping  us  to  gain  access  to  individuals  with  whom 
we  conducted  in-depth  interviews  and  ultimately  surveyed. 

Most  of  the  full-scale  interview  data  were  collected  during  the  sec¬ 
ond  and  third  sets  of  meetings  at  Warner  Robins  ALC.  In  the  second 
set  of  meetings  (April  7-8,  1998),  we  met  with  CRT  members  in  LF, 
LR,  LN,  LK,  LU,  and  LY.  In  the  third  set  of  meetings  (June  8-9,  1988), 
we  met  with  CRT  members  in  LN  and  LU.  The  meetings  took  place 
in  groups  to  minimize  ALC  time  and  effort.  We  interviewed  program 
managers,  procurement  contracting  officers,  production  manage¬ 
ment  specialists,  and  item  management  specialists.  Participants 
were  given  a  list  of  questions  before  the  meeting.  (Appendix  B  pre¬ 
sents  the  interview  questions.)  We  asked  interviewees  to  think  about 
the  questions  within  the  context  of  a  specific  contract  the  center  con¬ 
sidered  most  innovative  or  successful  in  incorporating  CREP  tenets. 

We  then  asked  a  series  of  questions  about  the  contracts  they  had  in 
mind,  covering  such  areas  as: 

•  Background  of  the  contract:  general  description  and  charac¬ 
teristics 


Contracting  process:  differences  in  the  process  as  a  result  of 
CREP 
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•  CRT/IPT:  how  teams  were  created,  who  participated,  how  the 
teams  worked  together 

•  Design  of  the  contract:  innovations  considered  and  incorporated; 
challenges  and  methods  of  resolution 

•  Lessons  learned:  what  the  CRT  learned,  what  individual  team 
members  learned,  how  lessons  were  transmitted  to  other  teams. 

Individuals  discussed  their  experiences  with  particular  CREP 
contracts.  Using  the  context  of  a  specific  contract,  we  asked  ques¬ 
tions  or  guided  discussions  toward  topics  related  to  organizational 
levers.  Individuals  told  us  what  worked  and  what  was  most  difficult 
in  incorporating  CREP  tenets.  The  information  from  these  inter¬ 
views,  along  with  the  organizational  levers  identified  in  the  literature 
review,  was  used  to  develop  the  survey  questions. 

We  next  discuss  the  survey  conducted  at  Warner  Robins  ALC.  The 
survey  asked  members  of  CRTs  how  strongly  they  agreed  with  a 
series  of  statements  about  acquisition  reform,  leadership,  per¬ 
formance  evaluation  and  rewards,  teaming  and  partnering,  and 
training  and  career  development.  (The  survey  questions  are  repro¬ 
duced  in  Appendix  C.) 

SURVEY  OF  WARNER  ROBINS  ALC 

Data  were  collected  on  organizational  levers  through  a  survey  of  key 
members  of  contract  repair  teams.  To  develop  the  survey,  we  relied 
on  the  aforementioned  literature  review  findings  and  on  in-depth  in¬ 
terviews  conducted  with  CRTs  identified  by  Warner  Robins  ALC  as 
on  the  cutting  edge  of  writing  innovative  contracts.  The  survey  in¬ 
cluded  those  elements  considered  in  the  literature  as  important  in 
eliciting  new  behaviors  from  individuals  in  private- sector  organiza¬ 
tions.  For  the  survey,  we  adapted  these  elements  to  a  CREP  context 
and  emphasized  issues  considered  as  key  drivers  or  inhibitors  of  in¬ 
novation  by  the  individuals  interviewed  at  the  ALC.  Warner  Robins 
ALC  senior  leadership  reviewed  the  survey  questions;  we  also  ran  a 
pilot  test  with  a  few  people  who  worked  on  CRTs  to  identify  potential 
wording  or  interpretation  problems.  (The  survey  questions  are  in 
Appendix  C.) 
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Conducted  over  two  periods — in  1998  and  1999 — the  web-based  sur¬ 
vey  asked  questions  on  perceptions  of  acquisition  reform  and  its  im¬ 
plementation  as  well  as  questions  concerning  individual  behavior 
and  incentives.  The  survey  was  distributed  to  four  positions  in  each 
CRT  that  the  teams  themselves  considered  the  most  significant  in  the 
design  and  writing  of  contracts — the  program  manager  or  logistics 
officer  (PM/LO),  the  item  management  specialist  (IMS),  the  pro¬ 
curement  contracting  officer  (PCO),  and  the  production  manage¬ 
ment  specialist-seller  (PMS).  Each  person  who  participated  provided 
only  one  set  of  responses. 

These  four  CRT  member  positions  play  critical  roles  in  the  contract 
process.  The  program  manager,  for  instance,  heads  the  CRT  and  is 
responsible  for  managing  weapon  systems  assets.  The  IMS  is  re¬ 
sponsible  for  estimating  requirements,  such  as  the  number  of  repairs 
for  a  particular  group  of  items  over  a  time  horizon.  Repair  produc¬ 
tion-related  issues  during  contract  execution  fall  to  the  PMS,  who 
works  from  the  ALC  side  to  anticipate  and  resolve  issues  that  arise 
during  repair  actions.1  PCOs  understand  contracting  policy  and 
regulations  and  actually  write  the  contract.  They  prepare  documen¬ 
tation  to  support  contract  review,  write  the  contract,  negotiate  the 
terms  and  conditions,  translate  various  decisions  and  agreements 
into  legally  binding  language,  and  help  select  the  contractors. 

Overall,  the  response  rate  for  the  survey  was  64  percent.2  Although 
the  survey  went  to  four  positions  per  contract  team,  many  of  the  in¬ 
dividuals  in  those  positions  sat  on  more  than  one  CRT  and  were  as¬ 
sociated  with  more  than  one  contract.  This  phenomenon  had  the 
effect  of  smoothing  out  some  of  the  variation  in  the  independent 
variables,  because  the  responses  of  an  individual  could  apply  to 
more  than  one  contract.  In  1998,  we  received  responses  from  CRTs 


during  CREP,  if  ALC-owned  and  operated  organic  shops  also  conducted  some  of  the 
repair  workload,  two  PMSs  participated:  the  buyer  PMS,  who  worked  with  the  IMS  to 
establish  the  requirement  for  contract  repair,  and  the  seller  PMS,  who  worked  with  the 
PCO  to  construct  a  contract.  Because  it  focused  on  CREP,  this  study's  survey  involved 
only  seller  PMSs. 

2The  sample  consisted  of  256  individuals:  66  PCOs,  40  PMSs,  75  PMs,  and  75  IMSs.  A 
breakdown  by  position  revealed  a  74  percent  response  rate  from  PCOs  (n  =  49),  a  63 
percent  response  rate  from  PMSs  (n  =  25),  a  60  percent  response  rate  from  PMs  (n  = 
45)  and  a  60  percent  response  rate  from  IMSs  (n  =  45).  Data  were  analyzed  from  79 
unique  CRTs  that  wrote  101  contracts. 
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responsible  for  84  contracts  written  primarily  in  1997  and  1998.  In 
1999,  we  again  surveyed  some  of  the  CRT  members  whose  first 
responses  contained  missing  data.  Other  members  who  had  not  re¬ 
sponded  at  all  were  again  asked  to  participate.  By  asking  CRT  mem¬ 
bers  to  complete  their  surveys  or  participate  if  they  had  not  re¬ 
sponded  in  the  first  survey,  the  number  of  contracts  in  our  sample 
increased  from  84  to  101.  Expanding  the  sample  size  to  101  contracts 
or  observations  increased  the  likelihood  the  regression  analyses 
would  detect  significance  in  levers  related  to  tenet  implementation. 
The  number  of  contracts  in  our  sample  was  an  important  constraint 
in  the  analyses,  because  it  limited  the  number  of  independent  vari¬ 
ables  used  in  the  regression  analyses.3 

ORGANIZATIONAL  LEVERS  (INDEPENDENT  VARIABLES) 
IDENTIFIED  PRIMARILY  THROUGH  PRINCIPAL 
COMPONENTS  ANALYSES 

We  analyzed  survey  responses  using  a  technique  called  Principal 
Components  Analysis  that  allows  one  to  examine  relationships 
among  item  responses  to  determine  whether  particular  questions 
reflect  the  same  underlying  concept.  It  is  a  general  method  of  devel¬ 
oping  scales  in  the  social  sciences,  used  for  example  in  personality 
and  intelligence  scales.4  Specifically,  the  PCA  technique  enabled  us 
to  construct  eight  organizational  lever  variables  from  responses  to 
survey  questions  that  were  closely  aligned  and  addressed  a  common 
theme.  We  first  describe  PCA  and  the  eight  independent  variables 
that  emerged  from  that  process.  Next  we  describe  two  independent 
variables  that  were  computed  directly.  At  the  end  of  this  section,  we 
discuss  the  final  steps  in  constructing  the  independent  variables. 
Appendix  C  shows  the  original  survey,  and  the  next  section  discusses 
those  questions  that  made  up  the  independent  variables. 


o 

We  would  have  liked  to  include  additional  variables  in  the  regression  models  to  help 
control  for  potential  systematic  bias  in  the  survey  responses,  e.g.,  dummy  variables  for 
job  title  of  respondents,  etc.;  unfortunately,  the  small  size  of  our  sample  limited  the 
addition  of  such  variables. 

4See  B.  G.  Tabachnick  and  L.  S.  Fidell,  Using  Multivariate  Statistics,  Third  Edition, 
New  York:  Harper  Collins,  1996. 
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PCA  showed  that  the  survey  questions  consisted  of  eight  separate 
themes  describing  organizational  levers.  These  levers  were  named: 
attitude  toward  acquisition  reform,  leadership  consistency,  perfor¬ 
mance  evaluation,  performance  incentives,  effective  teaming, 
contractor  partnering,  Air  Force  partnering,  and  training  in 
acquisition  reform.  Details  of  the  PCA  can  be  found  in  Appendix  D. 
Additionally,  the  survey  questions  used  to  construct  each  organiza¬ 
tional  lever  variable  are  provided  at  the  end  of  each  variable’s 
description  below. 

Attitude  Toward  Acquisition  Reform 

During  the  early  part  of  the  acquisition  reform  initiative  in  1996  and 
1997,  contracting  personnel  throughout  the  Department  of  Defense 
communicated  a  belief  that  reform  was  generally  a  good  idea,  but 
perhaps  not  for  operating-line  personnel,  particularly  if  better  con¬ 
tracts  meant  the  individual’s  workload  would  increase  and  fewer 
people  would  be  needed  to  do  the  work  in  the  future.  Some  people 
perceived  that  contract  outcome  improvements  might  jeopardize 
their  jobs.  Still  others  saw  benefits  in  solving  certain  problems  to  free 
up  their  time  to  solve  harder  problems. 

The  PCA  technique  identified  two  related  survey  questions  that 
asked  about  support  for  acquisition  reform.  The  first  question  asked 
about  the  extent  to  which  the  person  supported  the  goals  of  acquisi¬ 
tion  reform.  The  second  question,  with  five  subparts,  asked  about 
the  person’s  perception  of  how  acquisition  reform  would  affect  his  or 
her  contracts  and  job.  The  questions  were: 

•  How  supportive  are  you  of  the  goals  of  acquisition  reform?  (Q25) 

•  How  likely  is  acquisition  reform  to  contribute  to  the  following 
outcomes?  (Q26) 

•  Improve  mission  capability  rates  for  end  user 

•  Reduce  the  cost  of  my  contracts 

•  Increase  contractor  responsiveness/flexibility 

•  Increase  my  job  satisfaction 

•  Make  my  job  more  secure. 
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Responses  to  these  questions  formulated  the  attitude  organizational 
lever  variable. 

Leadership  Consistency 

AFMC  and  the  ALCs  are  under  tremendous  pressure  to  apply  best 
practices  where  it  makes  sense,  and  consequently  they  have  a  large 
number  of  initiatives  underway.  It  is  natural  for  operating-level  per¬ 
sonnel  to  work  more  aggressively  on  an  initiative  to  which  they  be¬ 
lieve  their  immediate  supervisor  and  senior  leadership  appear 
committed  and  that  they  expect  will  lead  to  permanent  change.  One 
would  expect  that  the  more  people  feel  leadership  is  consistently 
calling  them  to  enact  reform,  the  more  likely  they  will  do  so. 

Principal  components  analysis  indicated  that  two  parts  of  a  question 
on  leadership  support  were  related.  These  items  measured  the  ex¬ 
tent  to  which  the  person  felt  he  or  she  was  hearing  the  same  message 
about  reform  from  all  levels  of  management  and  whether  manage¬ 
ment  appeared  committed  to  reform  over  the  long  run.  The  ques¬ 
tions  were: 

•  How  true  are  the  following  statements  about  acquisition  reform? 
(Q28) 

•  Different  parts  of  the  chain  of  command  send  different 
messages  about  AR. 

•  Management  attitudes  suggest  that  AR  is  the  latest  “program 
of  the  month.” 

Performance  Evaluation 

The  business  and  management  science  literature  suggested  that  in¬ 
dividuals  perform  best  in  areas  in  which  they  are  evaluated  and  given 
incentives.  Thus,  if  the  success  of  acquisition  reform  depends  on 
better  teaming,  improved  contract  performance,  and  better  contrac¬ 
tor  relationships,  then  it  follows  that  performance  evaluation  criteria 
for  these  areas  are  important.  The  more  people  feel  they  will  be 
evaluated  on  reform  success,  the  more  likely  they  will  try  to  imple¬ 
ment  reform  innovation.  This  organizational  lever  must  be  viewed 
carefully,  because  the  CREP  initiative  was  still  ongoing  when  it  was 
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measured.  In  interviews,  we  were  told  personnel  evaluation  criteria 
had  not  changed  during  the  initiative,  so  we  expected  this  variable  to 
be  weakly  related  to  behavior  for  the  contracts  in  our  sample,  if  at  all. 
Still,  we  included  it,  because  of  its  importance  in  the  literature. 

PCA  indicated  four  subparts  of  a  question  about  performance  evalu¬ 
ations  were  related: 

•  How  important  are  the  following  factors  in  your  yearly  per¬ 
formance  evaluation?  (Q29) 

•  Improving  the  timeliness  of  contracts  processed  (reduced 
administrative  lead-time) 

•  Working  effectively  with  the  members  of  my  CRT 

•  Improving  contract  performance  (cost,  reliability,  cycle 
times) 

•  Improving  partnering  with  contractors. 

Performance  Incentives 

Unlike  performance  evaluation  criteria  that  already  exist  before  the 
initiative,  performance  incentives  often  develop  during  an  initiative, 
as  happened  at  Warner  Robins  ALC.  We  identified  performance  re¬ 
wards  used  by  the  private  sector  in  a  review  of  the  business  literature 
and  those  used  by  Warner  Robins  ALC  in  interviews  with  CRTs. 
Anecdotal  evidence  suggested  center  personnel  used  formal  recog¬ 
nition  from  the  organization  (e.g.,  plaques)  as  a  primary  incentive  for 
good  performance.  Several  individuals  we  interviewed  early  in  the 
study  indicated  positive  feedback  from  customers  (customer  surveys, 
letters  from  customers)  provided  an  effective  incentive  for  them  to 
continue  to  perform  well,  but  others  we  interviewed  thought 
voluntary  customer  recognition  (letters  or  phone  calls  from  cus¬ 
tomers)  was  too  subjective  and  arbitrary.  Some  of  the  individuals  we 
interviewed  expressed  concern  that  their  hard  work  was  not  appre¬ 
ciated  by  customers  or  their  immediate  supervisors  and,  as  a  conse¬ 
quence,  believed  this  lack  of  appreciation  undermined  their  enthusi¬ 
asm  for  taking  on  new  challenges  with  new  contracts.  We  expected 
to  see  more  innovation  as  people  perceived  they  would  be  rewarded 
for  their  success. 


22  Organizational  Levers  Can  Affect  Acquisition  Reform  Implementation 


PCA  found  a  relationship  between  items  in  a  five-part  survey  ques¬ 
tion  on  performance  incentives: 

•  How  likely  is  the  effective  use  of  AR  to  lead  to  the  following 
rewards?  (Q30) 

•  Higher  merit  increases 

•  Improved  opportunities  for  promotion 

•  Non-monetary  rewards  (e.g.,  time  off,  trips) 

•  Recognition  from  ALC/product  directorate/ division  of  a  job 
well  done 

•  Official  acknowledgment  of  customer  satisfaction. 

Effective  Teaming 

As  mentioned  earlier,  CREP  emphasized  the  concept  of  bringing  all 
key  members  together  early  in  the  contracting  process  to  decide  col¬ 
lectively  on  contract  goals  and  strategy.  This  teaming  concept  al¬ 
lowed  a  better  “give  and  take’’  with  functional  experts  who,  in  the 
past,  had  not  necessarily  understood  all  of  the  practical  and  legal 
constraints  by  which  other  members  were  bound.  CRTs  also  re¬ 
ported  to  us  in  interviews  an  appreciation  for  involving  the  contrac¬ 
tor  early  in  the  process.  In  some  cases,  CRTs  were  able  to  change  the 
contract  fundamentally  only  after  reviewing  the  contract  repair  pro¬ 
cess  with  the  contractor,  sharing  information  on  projected  require¬ 
ments  and  learning  from  the  contractor  about  particular  problems  in 
the  contract  repair  facility  to  arrive  at  fundamental  changes  in  the 
contract  together.  Effective  teaming,  however,  meant  that  the  team 
needed  to  overcome  functional  barriers,  such  as  understanding  pri¬ 
mary  objectives  and  constraints  in  each  step  of  the  process.  As  the 
team  formulated  a  common  strategy,  team  members  had  to  agree  to 
work  toward  a  common  performance  goal  and  include  early  on  any¬ 
one  who  had  potential  veto  power  over  the  proposed  contract.  We 
expected  the  relationship  between  effective  teaming  and  innovation 
to  be  positive,  i.e.,  as  effective  teaming  increased,  CRTs  would  be 
more  innovative. 

The  PCA  suggested  these  three  teaming  items  were  related: 
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•  How  true  are  the  following  statements  about  the  CRTs  (or  IPTs) 
you  have  worked  on?  (Q32) 

•  The  CRT  was  effective  in  overcoming  functional  or 
disciplinary  barriers  to  cooperation. 

•  Team  members  worked  toward  a  common  performance 
goal. 

•  Non-core  CRT  members  (FM,  BC,  CR,  EN,  QA)  were  well 
integrated  into  the  process. 

Contractor  Partnering 

Most  CRTs  reported  good  relationships  with  their  contractors  and 
most  had  worked  with  those  contractors  for  years.  CREP  set  as  its 
objective  improvements  in  repair  contract  outcomes,  which  necessi¬ 
tated  changing  expectations  (better  performance)  in  repair  contracts 
and  in  ways  of  working  with  contractors.  We  expected  that  innova¬ 
tion  would  occur  more  frequently  with  contractors  perceived  by 
team  members  to  be  contributing  positively  to  the  business  relation¬ 
ship.5 

The  PCA  technique  found  that  four  subquestions  about  contractor 
partnering  were  related: 

•  How  true  are  the  following  statements  about  the  relationship 
between  the  Air  Force  and  the  contractor?  (Q34) 

•  A  positive  working  relationship  exists  with  the  contractor. 

•  The  Air  Force  and  contractor  have  worked  together  to 
improve  processes  and  reduce  cycle  time. 

•  The  contractor  alerts  the  Air  Force  to  anticipated  problems 
that  could  affect  the  contractor’s  performance. 

•  The  contractor  shows  a  willingness  to  assume  risks  in  order 
to  do  business  with  the  Air  Force. 


innovation  can  certainly  occur  under  adversarial  conditions,  such  as  with  the  acqui¬ 
sition  of  new  systems  where  the  Air  Force  can  dictate  the  terms  unilaterally  when 
there  is  competition.  It  is  less  likely  to  occur  in  an  adversarial  relationship  with  a  sole- 
source  supplier.  Most  of  Air  Force  repair  contracts  are  with  sole-source  suppliers. 
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Air  Force  Partnering 

During  interviews  it  appeared  that  PMs  and  PCOs  were  particularly 
concerned  about  the  Air  Force’s  ability  to  be  a  better  customer  to  its 
suppliers.  In  particular,  they  were  concerned  about  its  willingness 
both  to  pay  for  the  risk  it  expected  the  contractor  to  assume  and  to 
reward  contractors  who  perform  well.  The  prevailing  approach  in 
repair  contracts  over  the  years  has  been  to  put  most  of  the  risk  on  the 
contractor  in  the  belief  that  it  would  minimize  costs,  because  the  Air 
Force  would  not  pay  for  any  workload  that  did  not  generate.  Under 
acquisition  reform  and  lean  logistics,  teams  found  they  had  to  as¬ 
sume  some  risk  up  front  to  get  the  decreases  in  the  repair  prices  and 
repair  cycle  times  they  wanted.  Thus,  we  expected  that  as  Air  Force 
partnering  increased  or  improved — measured  as  explicit  treatment 
of  risk  and  rewards — more  innovation  would  occur. 

PCA  indicated  two  subquestions  were  related  in  regard  to  Air  Force 
partnering: 

•  How  true  are  the  following  statements  about  the  relationship 
between  the  Air  Force  and  the  contractor?  (Q34) 

•  The  Air  Force  shows  a  willingness  to  pay  for  the  risks  it  wants 
the  contractor  to  assume. 

•  The  Air  Force  shows  a  commitment  to  reward  contractors 
who  perform  well. 

Training  in  Acquisition  Reform 

The  literature  explicitly  emphasizes  the  importance  of  training  em¬ 
ployees  in  how  to  integrate  new  practices  into  their  jobs.  Because 
the  survey  covered  several  topics  and  had  resource  constraints,  it 
asked  high-level  questions  only.  For  example,  it  measured  quantity 
of  training  days  but  not  the  quality  of  the  training  received.  In  this 
study,  we  expected  that  as  the  number  of  days  of  training  and  fre¬ 
quency  of  updates  people  received  increased,  innovation  would  also 
increase. 

Using  results  from  the  PCA  technique,  two  survey  questions  were 
used  to  construct  the  “training  in  acquisition  reform”  variable.  The 
first  question  asked  how  many  days  of  formal  training  in  acquisition 
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reform  a  person  received  in  the  past  two  years.  The  second  asked 
how  often  they  received  updates  in  acquisition  reform  training. 
These  questions  were: 

•  How  many  days  of  formal  training  in  acquisition  reform  have  you 
received  in  the  past  two  fiscal  years?  (Q41) 

•  How  often  do  you  receive  formal  and  informal  training  updates 
on  acquisition  reform  as  it  affects  your  job?  (Q42) 

Two  other  independent  variables  were  computed  directly.  The  ninth 
organizational  lever  variable — years  of  job  experience — used  data 
collected  in  the  survey  but  did  not  require  PCA.  The  tenth  variable — 
a  product  directorate  dummy  variable — also  did  not  require  PCA. 
We  describe  them  next. 

Job  Experience 

Work  experience  can  have  a  positive  and  negative  relationship  to  in¬ 
novation.  Long  experience  in  a  position  can  make  a  person  better 
informed  on  past  efforts  that  worked  or  failed.  Such  information  can 
make  someone  either  risk  averse  or  more  likely  to  succeed  by  know¬ 
ing  the  potential  pitfalls.  Lack  of  experience,  in  turn,  can  make  a  per¬ 
son  more  willing  to  try  something  new  and  challenging.  Anecdotal 
evidence  from  interviews  with  CRT  personnel  suggested  that 
younger  members  were  more  enthusiastic  and  willing  to  write  more 
innovative  contracts.  Therefore,  we  expected  those  with  less  job  ex¬ 
perience  would  be  more  likely  to  write  innovative  contracts. 

We  used  one  survey  question  to  construct  the  “job  experience’’  vari¬ 
able: 

•  How  many  years  of  work  experience  do  you  have  in  your  current 
job? (Q35-38) 

Product  Directorate 

At  the  time  of  this  study,  the  U-2  Product  Directorate  at  Warner 
Robins  used  large  sustainment  contracts  to  support  its  classified 
weapon  systems  instead  of  component  repair  contracts  used  by  the 
unclassified  programs.  These  sustainment  contracts  covered  a  vari- 
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ety  of  goods  and  services,  including  repair,  modifications,  engineer¬ 
ing,  new  parts  procurements,  and  scheduled  overhauls.  Because  of 
classification  requirements,  they  used  separate  data  systems  and 
slightly  different  contracting  rules  than  did  the  CRTs  in  the  other 
product  directorates.  Funds  on  U-2  contracts  generally  came  from 
appropriated  weapon  system- specific  funds  that  did  not  compete 
with  parts  from  other  weapon  systems,  as  do  items  that  use  revolving 
fund  dollars.  For  all  of  these  reasons,  we  expected  that  the  U-2 
Product  Directorate  had  more  opportunities  to  innovate  than  did  the 
other  product  directorates. 

We  tried  to  control  for  these  differences  in  contracts  with  a  dummy 
variable,  called  “Product  Directorate,”  analogous  to  a  commercial 
corporate  division.  If  the  CREP  contract  was  written  in  the  U-2 
Product  Directorate,  we  assigned  the  variable  a  value  of  1;  otherwise 
it  received  a  0. 

This  section  described  how  PCA  was  used  to  identify  organizational 
lever  themes  among  the  survey  responses  for  eight  independent 
variables  and  how  two  other  variables  were  computed.  We  next  dis¬ 
cuss  how  the  independent  variables  were  computed  for  each  organi¬ 
zational  lever  after  the  PCA  analyses. 

Independent  Variable  Computation 

PCA  indicated  themes  among  survey  questions  that  ultimately 
formed  eight  organizational  variables.  For  each  theme  identified  by 
PCA,  such  as  leadership  consistency,  a  respondent’s  values  were 
summed  across  all  of  the  questions  considered  by  PCA  as  related.6 
The  organizational  lever  variable  for  each  contract  was  determined 
by  averaging  the  values  of  individual  team  members  across  the  CRT. 

Because  each  member  provided  survey  responses  only  once,  re¬ 
sponses  sometimes  resulted  in  duplicates  if  an  individual  partici¬ 
pated  in  more  than  one  CRT  or  contract.  In  most  cases,  CRTs 
consisted  of  a  different  set  of  members,  thus  making  the  indepen- 


6Responses  to  survey  questions  took  on  values  that  ranged  from  1  to  5  (or  1  to  3,  in 
some  cases)  with  a  1  assigned  consistently  to  the  lowest  end  of  the  scale.  Thus,  re¬ 
sponses  to  survey  questions  associated  with  particular  organizational  lever  variables 
were  given  a  quantitative  value. 
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dent  variables  not  identical.  Ideally,  one  would  like  to  survey  CRT 
members  at  the  conclusion  of  each  contract  activity,  but  this  study 
was  allowed  to  conduct  the  survey  once  (the  second  survey  collected 
responses  from  CRT  members  who  had  not  responded  during  the 
first  year  of  data  collection  or  to  complete  missing  data  from  the  first 
survey).  The  survey  responses  represented  an  accumulated  experi¬ 
ence  for  CRT  members.  Response  rates  for  CRT  teams  varied  across 
contracts.  The  sample  included  contracts  that  had  survey  responses 
from  at  least  one  person  on  the  CRT.  Most  of  the  sample  contracts 
included  responses  from  two  or  more  CRT  members. 

We  also  evaluated  the  independent  variables  for  the  degree  to  which 
they  were  normally  distributed,  as  required  for  ordinary  least  squares 
(OLS)  regression  analysis.7  As  a  result,  we  transformed  two  vari¬ 
ables — attitude  toward  acquisition  reform  and  leadership  consis¬ 
tency — from  a  1  to  5  scale  to  a  1  to  3  scale  because  of  the  lack  of  vari¬ 
ability  across  the  5-point  scale.  Leadership  consistency  and  effective 
teaming  were  positively  skewed,  but  a  square  root  transformation  led 
to  a  negatively  skewed  distribution,  so  we  did  not  transform  these 
variables.  Distributions  across  the  rest  of  the  independent  variables 
were  not  highly  skewed. 

CREP  TENET  GROUPS  (DEPENDENT  VARIABLES) 

Now  we  turn  to  the  identification  of  our  dependent  variables.  For 
our  analysis,  the  dependent  variables  were  specific  groups  of  tenets 
derived  from  CREP  policy  objectives.  The  CREP  initiative  had  two 
primary  policy  objectives:  contract  innovations  consistent  with  ac¬ 
quisition  reform  and  innovations  consistent  with  agile  logistics.  The 
study  team  developed  dependent  variables  that  policymakers  could 
interpret  strategically.  Our  study  sponsor  was  interested  in  explain¬ 
ing  strategic,  policy-level  outcomes,  not  tactical  outcomes  such  as 
the  implementation  of  individual  tenets.  Once  we  decided  to  use 
groups  of  CREP  tenets  as  dependent  variables,  we  then  faced  the  is¬ 
sue  of  how  to  develop  the  groups.  We  initially  conducted  a  cluster 
analysis  of  the  CREP  tenets  in  our  sample  of  101  contracts  to  identify 
empirically  groups  of  CREP  tenets.  The  cluster  analysis  identified 


7We  tested  the  need  for  transforming  variables  with  the  PROC  UNIVARIATE  function 
in  SAS  system  software. 
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tenet  groups  that  had  no  meaningful  policy-relevant  theme,  which 
made  these  groupings  unusable.  In  the  end,  we  decided  to  formulate 
tenet  groups  based  on  policy  themes  highlighted  in  the  CREP  initia¬ 
tive  itself  and  in  a  policy  emphasized  at  the  beginning  of  the  initia¬ 
tive.8  We  next  consider  these  policy  themes  in  more  detail. 

CREP  Tenets:  Elements  of  the  Dependent  Variable 

As  noted  earlier,  during  CREP  implementation,  HQ  AFMC  provided 
its  ALCs  with  a  16-item  checklist  to  use  when  considering  tenets  to 
incorporate  in  each  CREP  repair  contract.  For  our  regression  analy¬ 
sis,  we  used  these  16  CREP  tenets  in  formulating  our  dependent  vari¬ 
ables.  HQ  AFMC  monitored  the  use  of  this  checklist,  and  each  ALC 
reported  the  actual  number  of  tenets  incorporated  in  contracts  on  a 
quarterly  basis.  This  information  was  the  data  source  for  the  depen¬ 
dent  variables  for  most  of  the  contracts  in  our  sample;  that  is,  these 
reports  became  the  basis  for  our  determination  of  the  extent  of  tenet 
incorporation.  Eight  contracts  in  the  sample  were  written  after 
September  1998,  and  thus  the  reporting  of  their  tenet  incorporation 
came  directly  from  Warner  Robins  ALC. 

The  CREP  tenets  used  in  the  analyses  are  listed  and  defined  below.9 
The  tenets  involve  acquisition  reform  and  agile  logistics-related  con¬ 
cepts.  When  necessary,  we  provide  definitions  commonly  accepted 
by  the  logistics  community. 

•  Reduce  contract  repair  prices:  lower  repair  prices  from  those  of 
the  same  item  in  the  previous  contract. 

•  Reduce  repair  cycle  time:  shorten  repair  cycle  time — the  time  an 
item  arrives  at  the  contractor  repair  facility  to  when  it  leaves — 
from  that  in  the  previous  contract. 


8At  the  beginning  of  the  CREP  initiative,  Warner  Robins  ALC  tried  to  implement 
as  many  CREP  tenets  as  possible  in  all  candidate  CREP  contracts,  which  meant  only 
the  easiest  tenets  could  be  incorporated  in  still-active  contracts.  These  easy-to- 
incorporate  tenets  were  related  to  fast  transportation  and  did  not  affect  any  of  the 
other  parts  of  the  contract.  These  tenets  are  included  in  the  “simple  modifications” 
tenet  group. 

9CREP  tenet  heading  language  came  from  Warner  Robins  ALC,  circa  1998. 
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Reduce  administrative  process  days:  shorten  administrative  lead- 
time  or  the  time  it  takes  to  write  a  contract  from  the  very 
beginning. 

Reduce  inventories:  anticipated  decrease  in  new  spare  buy 
requirements  that  result  from  a  shorter  repair  cycle  time  or 
reductions  in  other  parts  of  the  repair  pipeline,  such  as  fast 
transportation. 

Repair  on  demand:  repair  demand  set  by  the  EXPRESS  system, 
which  tries  to  maximize  the  probability  that  all  Air  Force  bases 
meet  their  aircraft  availability  goals  through  its  computed  repair 
priorities  that  depend  on  demand  forecasts  across  a  planning 
horizon,  actual  asset  positions,  and  a  host  of  logistics  considera¬ 
tions. 

Establish  a  consolidated  repairable  inventory! consolidated 
serviceable  inventory  (CRI/CSI):  the  depot  stock  level,  as  com¬ 
puted  by  Readiness-Based  Leveling  (RBL)  or  some  other  method, 
which  puts  the  pool  of  centrally  held  serviceable  and  un¬ 
serviceable  assets  at  the  contractor  facility  rather  than  the 
managing  ALC. 

Direct  shipment:  recoverable  items  sent  directly  from  an  Air 
Force  base  to  a  contractor  and  sent  back  to  an  Air  Force  base 
without  passing  through  a  Defense  Logistics  Agency  serviceable 
warehouse  at  the  managing  ALC. 

Maximize  use  of  commercial  practices:  practices  used  by  private- 
sector  firms  with  their  commercial  customers. 

Performance-oriented  specifications:  use  of  a  statement  of  objec¬ 
tive  or  a  generalized  set  of  performance  goals,  rather  than  a 
detailed  set  of  work  instructions  that  are  specified  by  the  Air 
Force. 

Fast  transportation:  use  of  expedited  air  transportation,  as 
needed. 

Shipment  on  demand:  shipment  instructions  for  the  newly 
produced  serviceable  that  the  Air  Force  transmits  electronically 
to  the  contractor. 

Early  contractor  involvement:  mainly  used  with  sole-source 
contractors,  although  CRTs  occasionally  used  face-to-face 
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meetings  with  prospective  bidders  for  competitive  contracts. 
The  CRT  met  with  the  contractor  early  in  the  process  and 
involved  the  supplier  in  constructing  the  new  contract, 
developing  the  statement  of  objective,  and  so  forth. 

•  Long-duration  contract:  long-term  contracts,  defined  by  AFMC 
as  three  years  or  more. 

•  Reduce  military  specifications  and  standards:  technical  re¬ 
quirements  eliminated  without  increased  risk  to  the  Air  Force, 
potentially  allowing  contractors  to  use  commercial  specifications 
and  standards  and  reduce  contractor  costs. 

•  Eliminate  unnecessary  data  requirements:  reporting  require¬ 
ments  deemed  outdated,  duplicative,  or  underutilized  could  be 
eliminated,  thus  potentially  reducing  contractor  costs  and  repair 
prices. 

•  Contract  performance  incentives:  use  of  a  clause  that  rewards  or 
penalizes  performance  that  deviates  significantly  from  baseline 
expectations. 

Figure  2.1  indicates  the  percentage  of  contracts  that  report  using 
each  innovation.  More  than  half  the  contracts  in  the  sample  in¬ 
cluded  10  of  the  16  CREP  tenets,  with  most  claiming  long- duration 
contracts.  The  tenets  used  with  the  least  frequency  in  the  sample  of 
contracts  were  contract  performance  incentives,  shipment  on  de¬ 
mand,  and  reduced  inventories. 10  Twenty-six  percent  of  the  sample 
contracts  came  from  the  U-2  Product  Directorate,  which  we  discuss 
at  the  end  of  the  next  section  on  the  construction  of  the  model’s  de¬ 
pendent  variables. 


^Anecdotal  evidence  from  interviews  suggests  several  reasons  for  the  infrequent  in¬ 
corporation  of  these  tenets.  Performance  incentives  work  well  with  objective  methods 
of  measuring  performance  and  understanding  the  value  of  different  performance  cri¬ 
teria,  e.g.,  the  worth  of  getting  an  item  returned  in  the  five  days  versus  ten  days. 
Shipment  on  demand  occurs  routinely  now,  but  at  the  time  of  the  study  data  system 
changes  were  not  available  to  make  this  occur  routinely.  Finally,  the  benefits  of  re¬ 
duced  inventories  applies  only  to  the  future,  so  if  the  inventory  is  already  plentiful  for 
a  particular  item,  reducing  cycle  time  will  affect  responsiveness  but  not  inventory. 


Methodological  Approach  31 


RAHDMR171 1-2. 1 


1. 

Long-duration  contract 

2. 

Reduce  military  specifications  and  standards 

3. 

Early  contractor  involvement 

4. 

Fast  transportation 

5. 

Reduce  administrative  process  days 

6. 

Reduce  repair  cycle  time 

7. 

Eliminate  unnecessary  data  requirements 

8. 

Direct  shipment 

9. 

Maximize  use  of  commercial  practices 

10. 

Repair  on  demand 

11. 

Performance-oriented  specifications 

12. 

Establish  a  CSI/CRI  1 

13. 

Reduce  contract  repair  costs  | 

14. 

f\  Reduce  inventories 

15. 

f\  Ship  on  demand 

16. 

f\  Contract  performance  incentives  |  | 

0  20  40  60  80  100 


Percentage  of  contracts 

NOTES:  26  percent  of  the  contracts  are  from  the  U-2  Product  Directorate,  n  =  101 


Figure  2.1 — Over  50  Percent  of  the  Contracts  in  Our  Sample  Incorporated 
10  of  the  16  Innovations 


CREP  Tenet  Groups:  The  Dependent  Variables 

As  described  in  the  last  section,  we  grouped  the  CREP  tenets  into 
four  policy-related  groups  and  a  fifth  group  that  included  all  16  CREP 
tenets.  The  four  policy-related  groups  were:  simple  modifications, 
key  acquisition  reform  concepts,  complete  acquisition  reform 
concepts,  and  agile  logistics.  The  fifth  group — all  16  CREP  tenets — 
was  added  for  completeness  to  show  whether  tenets  for  both  acquisi¬ 
tion  reform  and  agile  logistics  policies  were  being  implemented. 
Each  of  the  101  sample  contracts  was  evaluated  according  to  these 
groups.  The  value  of  the  dependent  variable  was  the  total  number  of 
tenets  for  a  CREP  innovation  group  that  were  incorporated  in  a 
contract. 
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We  describe  the  four  primary  policy-relevant  tenet  groups  below  and 
provide  the  complete  set  of  tenets. 

•  Simple  modifications:  Tenets  that  can  be  added  to  active 
contracts,  usually  related  to  speeding  up  transportation. 

•  Key  acquisition  reform  concepts:  The  set  of  acquisition  reform- 
related  tenets  that  are  clearly  exhibited  at  the  front  of  the 
contract  (e.g.,  price  and  cycle  time)  and  are  relatively  easy  to 
measure. 

•  Complete  acquisition  reform  concepts:  All  acquisition  reform 
tenets  included  in  the  key  acquisition  reform  group,  plus  those 
reform  tenets  not  as  easy  to  measure  or  identify  consistently  (e.g., 
maximize  commercial  practices). 

•  Agile  logistics:  All  agile  logistics  or  lean  logistics  tenets  designed 
to  reduce  logistics  pipelines  by  speeding  up  the  repair  and  trans¬ 
portation  segments  of  the  pipeline  as  well  as  improving  other 
logistics  efficiencies  (e.g.,  establish  a  CSI/CRI). 

•  All  CREP  tenets:  These  16  tenets  can  also  be  found  in  the 
combined  set  of  complete  acquisition  reform  concepts  and  agile 
logistics. 

Table  2.1  provides  the  particular  tenets  included  in  the  first  four 
groups  of  innovation  policy  objectives  that  we  analyzed  as 
dependent  variables.  The  fifth  group  included  all  tenets.  Figure  2.2 
shows  histograms  of  the  variability  in  innovation  across  the  five 
innovation  groups  or  dependent  variables.11  Each  histogram  shows 
the  percentage  of  contracts  in  our  sample  that  included  at  least  one 
tenet  defined  by  our  innovation  group  shown  in  Table  2.1. 

One  can  observe  in  the  first  histogram  in  Figure  2.2  that  most  of 
the  contracts  in  the  sample  (n  =  101)  included  at  least  two  simple 
modifications  (that  is,  transportation-related)  tenets.12  The  second 


1  ^he  live  most  common  tenets  used  were  early  contractor  involvement,  long-term 
contracts,  reduction  or  elimination  of  military  specifications,  fast  transportation,  and 
the  elimination  of  unnecessary  data  requirements. 

12These  tenets  are  included  also  in  the  simple  modifications  and  agile  logistics  mod¬ 
els. 
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Table  2.1 

CREP  Tenets  Formed  Four  Primary  Types  of  Innovation  Groups 


Type  of  Innovation  Groups 


Simple 

Modifica- 

KeyAR 

Complete 

Agile 

Tenets 

tion 

Concepts 

AR 

Logistics 

Reduce  contract  repair  price 

Reduce  repair  cycle  time 


Reduce  administrative  process  days 

Repair  on  demand _ 

Establish  a  CSI/CRI _ 

Direct  shipment _ 

Max  use  of  commercial  practices 

Performance-oriented  specs _ 

Reduce  inventories _ 

Fast  transportation _ 

Ship  on  demand _ 

Early  contractor  involvement _ 

Long-duration  contract _ 

Reduce  mil  specs  and  stds _ 

Eliminate  unnecessary  data 

requirements _ 

Contract  perform  incentives 


histogram  shows  that  the  sample  of  contracts  defined  in  the  key 
acquisition  reform  concepts  tenet  group  primarily  incorporated 
between  two  and  four  acquisition  reform  tenets  that  were  more 
objective  to  measure.  The  histogram  labeled  as  complete  acquisition 
reform  shows  that  our  sample  of  contracts  typically  incorporated 
seven  or  eight  of  the  tenets  involving  acquisition  reform.  Most  of  the 
101  contracts  in  our  sample  incorporated  between  three  and  five 
agile  logistics  tenets,  shown  by  the  histogram  labeled  agile  logistics. 
Finally,  our  sample  of  contracts  incorporated  between  9  and  12  CREP 
tenets,  shown  in  the  last  histogram  labeled  all  CREP  tenets. 


34  Organizational  Levers  Can  Affect  Acquisition  Reform  Implementation 


®  <s  60 

a)  t3 

f  2  40  - 
o  o  20 

S.  o  0 


Simple  modifications 


L 


L 


a)  cn  60 
g>  O 


0  12  3 

Number  of  tenets 

Complete  acquisition  reform 


RAUDMR171 1-2.2 

Key  acquisition  reform  concepts 


®  “  60 
t5 

|  2  40  I— 

CD  £ 

a  8 

<D  y. 

0-  O 


1  II — 11 

ll - 1 

0  1  2 

3 

4 

5 

Number  of  tenets 
Agile  logistics 


- 

CD  tQ  60 

-2  2  40 

i_i_i_i_nni  J  in 

cd  ^  on 

sg20 

CD  JZ 

°  0 

Number  of  tenets 


Number  of  tenets 


All  CREP  tenets 


2  40  — 
8  o  20  - 


I  1 1 — ili — ii  li — ill — ill  111  111 


l~l  n  n  I~l 


ILiJ-l. 


0  2  4  6  8  10  12  14  16 
Number  of  tenets 


Figure  2.2 — Number  of  Tenets  Incorporated  into  CREP  Contracts 
by  Tenet  Group 


Several  tenets  appear  in  more  than  one  dependent  variable;  for  ex¬ 
ample,  the  reduction  in  repair  cycle  time  is  an  objective  shared  by 
both  acquisition  reform  and  agile  logistics  and  appears  in  both  de¬ 
pendent  variables.  Groups  of  tenets  allowed  us  to  ask  whether  or¬ 
ganizational  levers  could  help  explain  the  types  of  broad  policy  ob¬ 
jectives  Warner  Robins  ALC  set  for  CREP  contracts,  namely,  simple 
modifications,  acquisition  reform,  agile  logistics,  or  all  of  the  above. 

REGRESSION  ANALYSES 

Each  of  the  four  tenet  groups,  plus  the  fifth,  complete  set  of  tenets, 
became  a  separate  regression  model.  We  analyzed  the  same  set  of 
101  CREP  contracts  to  determine  the  relationship  organizational 
levers  had  with  the  incorporation  of  particular  tenets  in  each  of  our 


Methodological  Approach  35 


defined  tenet  groups.  According  to  information  learned  in  inter¬ 
views,  CRTs  were  instructed  by  AFMC  to  apply  as  many  of  the  CREP 
tenets  as  were  appropriate  in  each  contract. 

In  five  separate  multiple  regression  analyses,  we  regressed  the 
groups  of  innovative  contract  types  onto  the  nine  organizational 
levers  (see  pages  17  to  25)  and  a  U-2  Product  Directorate  dummy 
variable  (see  page  25).  The  dummy  variable  allowed  us  to  account 
for  any  systematic  differences  between  the  U-2  contracts  and  the 
other  75  percent  of  the  contracts  in  our  sample.13  Thus,  we  used 
regression  analysis  to  explain  how  organizational  levers  were  related 
to  reported  CREP  innovation.  A  detailed  description  of  the  regres¬ 
sion  analyses  and  their  results  can  be  found  in  Appendix  E.  We  dis¬ 
cuss  the  results  of  our  regression  analyses  in  the  next  chapter. 

SUMMARY  OF  METHODOLOGICAL  APPROACH 

As  this  chapter  highlights,  our  study  began  with  a  literature  review  on 
organizational  levers  and  innovation  in  the  private  sector.  We  used 
this  information  to  structure  interviews  with  personnel  considered  to 
have  excelled  at  incorporating  CREP  tenets  in  the  first  CREP  con¬ 
tracts.  The  literature  review  and  interviews  formed  the  basis  for  the 
survey  of  key  personnel  participating  on  contract  repair  teams,  i.e., 
program  managers,  procurement  contracting  officers,  production 
management  specialists,  and  item  management  specialists.  Using 
PCA,  we  identified  and  measured  organizational  levers  from  survey 
questions.  Using  AFMC  data,  we  classified  the  total  number  of  re¬ 
ported  CREP  tenets  into  five  tenet  groups:  simple  modifications,  key 
acquisition  reform  concepts,  complete  acquisition  reform  concepts, 
agile  logistics,  and  all  CREP  tenets.  We  then  measured  the  use  of 
these  tenets  in  each  of  the  101  contracts  in  our  sample.  Finally,  we 
regressed  these  CREP  tenet  groups  on  the  organizational  levers  to 
determine  the  individual  relationship  each  organizational  lever  had 


1  3 

We  refer  the  reader  to  Appendix  E  for  regression  analyses  for  CREP  contracts  minus 
the  U-2  contracts  (n  =  75  contracts).  The  resulting  set  of  models  had  less  explanatory 
power  for  acquisition  reform  policy  alternatives.  Excluding  U-2  contracts  reduces  the 
ratio  between  independent  or  predictor  variables  and  sample  size,  which  can  cause 
variables  to  drop  in  significance.  We  caution  the  reader  from  drawing  conclusions 
from  this  smaller  sample.  We  include  these  analyses  because  the  results  isolate  the  ef¬ 
fects  of  organizational  variables  on  CRTs  writing  component  repair  contracts. 
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with  the  number  of  CREP  tenets  incorporated  in  the  contract,  hold¬ 
ing  all  other  levers  constant.  The  results  of  the  regression  analyses 
are  discussed  in  the  next  chapter. 


Chapter  Three 

MAIN  FINDINGS  AND  IMPLICATIONS 


This  chapter  presents  the  results  of  our  regression  analyses  and  dis¬ 
cusses  their  primary  implications  for  Air  Force  policy.  The  analyses 
were  designed  to  measure  the  statistical  relationship  between  the 
different  organizational  levers  and  operating-level  personnel 
behavior.  For  a  complete  statistical  account  of  the  regression  results, 
we  refer  the  reader  to  Appendix  E,  which  describes  two  sets  of 
regression  analyses  of  the  innovation  groups,  one  containing  our 
complete  sample  of  101  Warner  Robins  ALC  contracts  and  another 
sample  set  of  only  75  Warner  ALC  contracts.  The  difference  between 
the  two  groups  is  that  the  former  includes  U-2  Product  Directorate 
sustainment  contracts,  which  have  repair  components,  whereas  the 
latter  contains  only  repair  contracts.  Of  the  two,  the  set  of  101 
contracts  has  greater  statistical  power  because  of  its  larger  sample 
size.  Thus,  the  discussion  below  considers  regression  analyses 
results  only  from  the  larger  sample. 

ORGANIZATIONAL  LEVERS  AND  THEIR  RELATIONSHIP 
WITH  INNOVATION 

Several  patterns  emerged  from  our  regression  analyses.  First,  the 
analyses  provide  empirical  evidence  that  organizational  levers  show 
significant  statistical  relationships  with  reported  contract  innova¬ 
tion.  People’s  behavior  in  incorporating  tenets  in  contracts  is  associ¬ 
ated  with  levers  such  as  attitude  toward  acquisition  reform,  effective 
teaming,  performance  evaluation,  and  so  on.  This  finding  shows  that 
policies  and  practices  related  to  personnel  behavior  were  a  factor  in 
the  extent  to  which  CRTs  incorporated  tenets  in  repair  contracts. 
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Second,  organizational  levers  appeared  most  statistically  related  to 
the  implementation  of  agile  logistics  tenets,  followed  by  simple 
modifications.  Conversely,  they  were  less  associated  with  acquisition 
reform  goals. 

Training  in  acquisition  reform  consistently  was  related  to  reported 
tenet  use.  Training  in  acquisition  reform  had  a  statistically  signifi¬ 
cant  positive  relationship  with  four  contract  innovation  groups  and 
the  group  of  all  CREP  tenets.  CRTs  who  reported  more  training  in 
acquisition  reform  also  incorporated  more  tenets.  At  the  highest 
level  of  significance  (p  <  0.01),  the  analyses  showed  that  as  CRTs  re¬ 
ported  receiving  more  training  in  acquisition  reform,  they  incorpo¬ 
rated  more  tenets  in  the  innovation  groups  of  simple  modifications, 
agile  logistics,  and  all  CREP  tenets.  At  a  slightly  lower  level  of  signifi¬ 
cance  (p  <  0.05),  CRTs  who  reported  more  training  in  acquisition  re¬ 
form  incorporated  more  tenets  in  the  complete  acquisition  reform 
innovation  group.  Finally,  at  a  still  lower  level  of  significance  (p  < 
0.10),  CRTs  who  reported  more  training  in  acquisition  reform  incor¬ 
porated  more  tenets  in  key  acquisition  reform  concepts.  Personnel 
on  CRTs  who  received  more  training  used  more  tenets  in  their  con¬ 
tracts,  compared  to  personnel  on  CRTs  with  less  training.  Because  of 
the  constraints  of  the  survey,  we  could  not  collect  information  on  the 
quality  of  the  training;  however,  given  the  consistent  relationship 
between  training  and  innovation  and  Air  Force  interest  in  further 
exploring  quality  in  training  and  innovation,  these  results  suggest 
that  additional  study  is  warranted. 

Effective  teaming  had  the  second  most  consistent  statistical  rela¬ 
tionship  with  contract  innovation  in  all  tenet  groups,  except  simple 
modifications.  However,  the  relationship  between  effective  teaming 
and  the  extent  of  innovation  for  these  groups  was  negative.  In  fact, 
effective  teaming  and  the  key  acquisition  reform  concepts  innova¬ 
tion  group  was  the  most  statistically  significant  relationship  (p  < 
0.01),  followed  by  agile  logistics  and  all  CREP  tenets  (p  <  0.05),  and 
then  by  complete  acquisition  reform  innovation  (p  <  0.10).  We  ex¬ 
pected  teams  that  considered  themselves  effective  would  write  more 
innovative  contracts.  Surprisingly,  the  opposite  relationship  was 
found.  The  analyses  indicate  that  as  CRTs  reported  more  problems 
with  teaming,  they  implemented  more  tenets  in  their  contracts  com¬ 
pared  with  teams  that  reported  higher  team  effectiveness  scores. 
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Although  not  a  statistically  significant  factor  in  terms  of  adopting  a 
simple  modifications  model,  the  negative  relationship  is  consistent. 

While  this  negative  relationship  seems  counterintuitive,  we  hypoth¬ 
esize  that  when  contracting  members  worked  as  a  team  under  CREP 
reform,  rather  than  as  individuals  under  the  old  process,  members 
had  to  make  consensus-like  decisions  in  a  new  way  that  required 
more  give  and  take.  Perhaps  teams  that  pushed  reform  further  pres¬ 
sured  its  members  to  learn  new  practices,  take  more  risks,  work 
longer  hours,  and  spend  more  time  trying  to  bring  around  the  skep¬ 
tics  among  its  members.  Perhaps  innovation  is  occurring  by  fiat  over 
the  objections  of  other  team  members,  leading  to  a  sense  that  the 
team  is  not  unified  in  purpose  or  direction.  On  the  other  hand, 
teams  that  reported  an  easier  teaming  experience  may  have  settled 
for  less  aggressive  goals.  In  retrospect,  it  is  not  surprising  that  the 
teaming  variable  was  not  related  to  the  simple  modification  goals, 
because  these  innovations,  which  are  simpler  to  incorporate  and 
change  little  of  the  basic  contractor-Air  Force  relationship,  do  not  in¬ 
volve  as  much  creative  problem  solving  among  team  members  as 
other  goals. 

A  study  conducted  by  Gerald  Miller  (1993)  may  further  explain  these 
counterintuitive  findings.  In  a  study  of  the  relationship  between 
team  stability  and  learning  among  work  teams,  Miller  found  that  un¬ 
stable  teams — teams  whose  members  changed  frequently — experi¬ 
enced  higher  learning  rates  when  confronted  with  new  problems. 
Work  team  instability  encouraged  members  to  expand  alternatives 
and  solutions.  He  also  found  that  unstable  teams  required  more 
complex  management  arrangements  to  keep  team  members  on 
track. 

The  negative  relationship  may  also  be  a  result  of  the  question  word¬ 
ing  in  the  survey.  The  survey  asked  about  end-state  conditions  of 
teaming  effectiveness,  such  as  the  effectiveness  of  the  CRT  to  over¬ 
come  barriers  to  cooperation  or  the  extent  to  which  team  members 
worked  toward  a  common  goal.  Constraints  on  our  survey  did  not 
allow  us  to  pursue  the  exact  nature  of  CRT  teamwork  processes  or 
attitude  measures,  such  as  group  dynamics.  Future  research  might 
clarify  this. 
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Ultimately,  the  negative  effective  teaming  result  was  perhaps  our 
most  surprising  and  potentially  troublesome  finding.  If  these  nega¬ 
tive  experiences  continue — and  we  do  not  know  if  they  have  because 
this  study  ended  not  long  after  the  end  of  the  CREP  initiative — teams 
may  opt  for  less  innovation  if  smoother  team  dynamics  are  preferred. 
This  result  strongly  suggests  that  AFMC  and  the  ALCs  will  want  to  in¬ 
vestigate  teaming  more  thoroughly. 

Attitude  toward  acquisition  reform  had  the  third  most  consistent 
statistical  relationship  with  contract  innovation.  It  had  a  statistically 
significant  positive  relationship  in  each  innovation  group  except  for 
key  acquisition  reform  concepts.  The  regression  analyses  showed, 
with  a  high  level  of  significance  (p  <  0.05),  a  positive  relationship  be¬ 
tween  attitude  toward  acquisition  reform  and  simple  modifications 
and  all  CREP  tenets  innovation  groups.  A  positive  relationship  be¬ 
tween  team  member  attitude  toward  acquisition  reform  and  the  in¬ 
corporation  of  tenets  in  the  complete  acquisition  reform  and  agile 
logistics  innovation  groups  had  a  slightly  lower  level  of  significance 
(p  <  0.10).  These  results  suggest  that  those  CRTs  that  view  reform 
more  positively  are  associated  with  greater  implementation  of  re¬ 
form  tenets  in  their  contracts.  One  can  also  think  about  attitude  as 
an  approximate  measure  of  operating- level  “buy-in.”  Those  teams 
that  agree  with  the  goals  of  the  initiative  are  associated  with  more 
innovative  contracts. 

Contractor  partnering  showed  a  positive  relationship  with  tenets 
implemented  for  simple  modifications  and  agile  logistics  goals.  This 
result  indicates  that  those  CRTs  that  included  more  reform  tenets  in 
contracts  also  perceived  contractors  to  be  better  partners.  The  signif¬ 
icance  level  for  the  positive  relationship  between  contractor  partner¬ 
ing  and  the  incorporation  of  tenets  in  an  innovation  group  was  high¬ 
est  with  simple  modifications  (p  <  0.01)  and  slightly  less  with  agile 
logistics  (p  <  0.05).  This  result  is  encouraging,  but  the  relationship  is 
not  statistically  significant  for  either  of  the  two  acquisition  reform 
models  or  for  the  model  including  all  CREP  tenets. 

Leadership  consistency  was  statistically  significant  only  in  the  agile 
logistics  (p  <  0.01)  and  key  acquisition  reform  (p  <  0.05)  innovation 
models.  Leadership  consistency  had  a  positive,  significant  relation¬ 
ship  with  CRT  incorporation  of  agile  logistics  and  key  acquisition 
reform  tenets.  CRTs  that  perceived  consistent  messages  throughout 
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management  and  that  believed  reform  would  be  around  for  some 
time  were  associated  with  the  incorporation  of  these  types  of  tenets 
in  their  contracts.  According  to  Larkin  and  Larkin  (1994),  rank-and- 
file  employees  listen  to  their  first-level  supervisors  and  not  just  se¬ 
nior  leadership.  These  employees  need  to  hear  about  change  repeat¬ 
edly  from  their  supervisors  before  they  believe  the  change  effort  will 
have  staying  power.  If  supervisors  are  not  committed  to  change, 
Larkin  and  Larkin  found  it  is  very  difficult  for  them  to  win  over  rank- 
and-file  employees.  The  survey  in  our  study  did  not  ask  questions 
about  middle  management  as  an  organizational  lever,  but  the  results 
suggest  it  warrants  further  consideration. 

Performance  evaluation  had  a  negative  statistical  relationship  with 
agile  logistics  tenets  (p  <  0.05).  One  would  typically  expect  this  or¬ 
ganizational  lever  to  have  a  positive  relationship  with  innovation,  so 
we  found  these  results  surprising  and  deserving  of  attention.  The 
analysis  indicates  that  those  CRTs  that  said  they  were  more  likely  to 
be  graded  on  improving  the  timeliness  of  the  contracts  they  pro¬ 
cessed,  working  effectively  with  other  team  members,  improving 
contract  performance,  and  improving  partnering  with  contractors 
were  less  likely  to  incorporate  agile  logistics  reform  tenets.  In  addi¬ 
tion,  our  results  indicate  no  relationship  between  performance  eval¬ 
uation  and  the  other  tenet  models.  To  explain  these  results,  we  hy¬ 
pothesize  that  those  CRT  members  who  are  more  likely  to  succeed  in 
writing  innovative  contracts  feel  they  will  not  be  rewarded  suffi¬ 
ciently  for  their  effort.1  If  correct,  it  suggests  that  the  ALCs  will  need 
to  align  performance  evaluation  criteria  more  closely  to  desired  new 
behavior,  to  apply  perhaps  to  all  team  members. 

Job  experience  showed  a  negative  statistical  significance  for  the  sim¬ 
ple  modifications  and  complete  acquisition  reform  innovation 
groups  (p  <  0.10).  It  had  no  relationship  with  the  other  two  innova¬ 
tion  groups  or  the  complete  set  of  tenets.  The  negative  relationship 
indicates  that  CRTs  included  slightly  more  transportation-related 
(simple  modifications)  tenets  to  contracts  as  the  level  of  personnel 
experience  decreased.  However,  the  magnitude  of  its  relationship 


^ur  results  suggest  that  the  ALCs  might  want  to  align  performance  incentives  with 
other  contracting-related  initiatives,  if  they  have  not  done  so  already,  and  include 
criteria  explicitly  designed  to  reinforce  CRT  contract  innovation. 
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with  the  incorporation  of  reform  tenets  is  so  slight  as  to  have  little 
practical  consequence. 

Two  variables — performance  incentives  and  Air  Force  partnering — 
had  no  explanatory  power  in  any  of  the  models  we  tested.  Senior 
leadership  indicated  Warner  Robins  ALC  has  systematically  and  in¬ 
tensively  tried  to  provide  nonmonetary  rewards  for  writing  particu¬ 
larly  innovative  contracts.  Perhaps  the  types  of  incentives  specified 
in  our  survey  were  not  sufficiently  representative  of  rewards  offered 
or  perhaps  personnel  do  not  perceive  these  rewards  as  effective.  We 
were  somewhat  surprised  by  the  lack  of  explanatory  power  for  Air 
Force  partnering.  It  suggests  that  there  is  no  consistent  relationship 
between  the  role  of  the  Air  Force  as  customer  and  the  number  of  in¬ 
novative  tenets  included  in  repair  contracts.  This  study  could  not 
delve  into  the  complex  issues  involved  in  how  the  Air  Force  works 
with  its  suppliers.  More  attention  needs  to  be  given  to  this  important 
issue.  Many  ongoing  transformation  initiatives,  such  as  purchasing 
and  supply  chain  management  (PSCM),  corporate  contracting,  and 
strategic  sourcing,  depend  on  effective  contractor  and  Air  Force 
partnering. 

Finally,  the  U-2  Product  Directorate  dummy  variable  showed  statis¬ 
tical  significance  with  every  innovation  group  except  simple  modifi¬ 
cations.  We  included  this  variable  to  account  for  the  important  dif¬ 
ferences  between  sustainment  and  component  repair  contracts  and 
their  respective  contracting  processes  and  environment.  The 
dummy  variable  accounts  generally  for  binary  differences  within  a 
sample  set,  in  this  case,  sustainment  versus  repair  contracts. 
Holding  all  other  organizational  levers  constant,  the  dummy  variable 
showed  that  the  U-2  directorate  incorporated  more  tenets  in  the 
complete  acquisition  reform,  agile  logistics,  and  all  CREP  tenet 
groups  (p  <  0.01).  The  U-2  directorate  also  incorporated  more 
key  acquisition  reform  tenets  with  a  slightly  lower  significance  level 
(p  <  0.05).  At  the  time  of  this  study,  the  U-2  Product  Directorate 
reported  a  high  rate  of  tenet  use,  which  has  been  substantiated  in 
follow-up  discussions  at  Warner  Robins  ALC. 

LESSONS  FOR  THE  AIR  FORCE 

In  general,  the  results  suggest  that  the  Air  Force  should  continue  to 
make  effective  use  of  those  organizational  levers  associated  with 
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positive  results,  especially  training  in  and  fostering  positive  attitudes 
toward  acquisition  reform.  In  turn,  it  should  revisit  the  levers  associ¬ 
ated  with  null  or  negative  relationships,  especially  performance 
evaluation  and  teaming.  Specifically,  the  Air  Force  could  do  more  to 

•  Inform  personnel  that  senior  leadership  wants  to  see  progress  in 
achieving  well-specified  contracting  goals. 

•  Learn  how  to  create  effective  teams  through  training  in  group 
problem-solving  and  working  with  others  from  different 
functional  backgrounds,  in  addition  to  educating  teams  on  legal 
and  policy  changes. 

•  Align  personal  evaluation  criteria  and  incentives  to  reform  goals. 


EXTENDING  THESE  RESULTS  TO  OTHER  AIR  FORCE 
INITIATIVES 

To  the  degree  that  the  CREP  initiative  is  representative  of  contract 
reform  efforts  in  general,  the  Air  Force  can  reinforce  the  organiza¬ 
tional  levers  related  to  innovation  and  look  closely  at  those  that  seem 
unrelated  or  are  negatively  related. 

Are  the  lessons  from  this  study  applicable  to  other  ALCs?  The  answer 
depends  on  how  similar  the  CREP  initiative  at  Warner  Robins  ALC  is, 
in  terms  of  the  organizational  context  and  policy  goals,  to  other  ini¬ 
tiatives  and  at  other  locations.  We  hypothesize  that  it  is  reasonable 
to  think  the  relationship  between  organizational  levers  and  contract 
innovation  at  Warner  Robins  ALC  is  similar  to  the  relationship  of 
these  variables  at  both  Oklahoma  City  and  Ogden  ALCs.  However,  it 
is  also  possible,  because  this  study  analyzed  contracts  at  a  center 
viewed  by  HQ  AFMC  as  innovative,  that  these  results  would  not  ap¬ 
ply  to  the  other  two  centers.  Also,  senior  leadership  at  the  other 
ALCs  may  have  had  a  different  impact  on  personnel  behavior  at  their 
centers.  One  would  have  to  conduct  a  similar  analysis  at  these  cen¬ 
ters  to  know  conclusively  whether  organizational  levers  could  help 
explain  innovation  at  those  sites.2 


The  inclusion  of  those  two  centers  probably  would  have  increased  the  sample  size 
enough  to  allow  for  dummy  variables  to  represent  the  respective  centers.  We  tried  at 
one  point  to  expand  this  study  to  include  contracts  written  at  other  Air  Logistics 
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Could  the  same  relationship  between  levers  and  innovation  that  we 
find  under  the  CREP  initiative  occur  with  other  contract  initiatives? 
After  all,  the  Air  Force  has  taken  on  other  contracting-related  initia¬ 
tives  since  CREP,  many  of  which  attempt  more  significant  change  in 
behaviors  and  even  organizational  structure,  such  as  the  PSCM  ini¬ 
tiative.3  If  senior  leadership  does  not  address  the  levers  identified  as 
having  no  relationship  or  having  a  negative  relationship  with  inno¬ 
vation  implementation,  then  leadership  may  have  to  work  as  dili¬ 
gently — or  more  diligently,  depending  on  the  initiative — to  imple¬ 
ment  new  innovative  practices,  much  as  it  did  for  CREP. 

This  study  should  help  leadership  consider  the  behavioral  implica¬ 
tions  of  contracting  initiatives  such  as  CREP  and  decide  what  to  do  if 
it  wants  personnel  to  implement  contract  innovation.  This  study 
also  provides  an  analytical  approach  to  measuring  the  relationship 
between  organizational  levers  and  forms  of  contract  innovation 
other  than  CREP.  Since  this  study  began,  the  Air  Force  has  added 
initiatives  that  are  more  complicated  than  CREP  in  their  approach 
and  expected  outcomes,  such  as  PSCM,  corporate  contracting,  and 
performance-based  contracting.  The  need  to  encourage  new  ap¬ 
proaches  and  reinforce  desired  behavior  at  the  ALCs  is  important  to 
the  success  of  the  Air  Force’s  ongoing  transformation  efforts — a  pro¬ 
cess  in  which  support  at  the  operating  level  is  essential.  As  such,  the 
Air  Force  may  want  to  apply  this  methodology  to  other  initiatives 
where  behavioral  changes  are  needed  to  fulfill  larger  transforma¬ 
tional  goals. 


Centers  to  capture  potential  item  and  cultural  differences  at  Oklahoma  City  and 
Ogden  ALCs,  but  repair  workload  backlogs  in  1998  and  1999  prevented  the  other  ALCs 
from  participating. 

O 

The  PSCM  initiative,  demonstrated  on  the  F100  engine  at  Oklahoma  City  ALC  in 
FY2002,  has  as  its  objective  a  strategic  means  of  selecting  and  managing  suppliers  to 
provide  more  effective  and  efficient  support  to  the  warfighter.  While  consistent  with 
CREP  outcome  goals,  it  would  work  with  purchasing  and  supply  chain  activities  from 
an  enterprise-level  to  an  operating-level  perspective.  Behavioral  implications  for 
PSCM  are  more  significant  than  for  CREP. 


Appendix  A 

REPAIR  PRICE  AND  NEGOTIATED  FLOW  DAY 
DIFFERENCES  IN  WARNER  ROBINS  ALC  CREP 
AND  NON-CREP  CONTRACTS 


In  a  separate  analysis  that  complements  this  study,  we  examined 
whether  CREP  made  a  difference  in  contract  performance  outcomes, 
specifically,  in  repair  prices  and  negotiated  flow  days  (the  repair  cy¬ 
cle  time  required  by  a  contract) . 1  The  question  of  whether  the  initia¬ 
tive  leads  to  a  measurable  improvement  in  outcomes  was  of  interest 
to  the  Air  Force.  The  results  described  in  this  Appendix  suggest  that 
CREP  repair  contracts  written  by  Warner  Robins  ALC  during  the  ini¬ 
tial  phase  of  CREP  reduced  repair  cycle  time  without  increasing  unit 
repair  prices. 

The  analysis  described  here  collected  data  on  CREP  and  non-CREP 
contracts,  i.e.,  contracts  written  during  the  same  period  as  CREP  but 
that  did  not  use  the  CREP  process.  We  also  collected  data  on  con¬ 
tracts  directly  preceding  CREP  to  compare  item  prices  and  negoti¬ 
ated  flow  days  on  CREP-period  contracts  with  prices  and  flow  days 
for  the  same  items  on  the  just-expired,  old  contracts.  This  method 
allowed  us  to  control  for  technical  differences  across  items  while 
comparing  changes  in  price  and  flow  days. 

In  sum,  we  found  that  CREP  contracts  reduced  flow  days  without  in¬ 
creases  to  unit  repair  price  in  comparison  with  non-CREP  contracts 
written  in  the  same  period.  Non-CREP  contracts  also  had  reduced 


^Negotiated  flow  days  or  repair  cycle  time  is  the  time  from  an  item’s  induction  into 
repair  to  its  completed  repair. 
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flow  days  compared  with  previous  contracts,  but  unit  repair  prices 
increased.  The  statistical  analyses  that  follow  help  to  explain  these 
findings. 

Data  from  the  Contract  Depot  Maintenance  Production  and  Cost 
System  (G072D)  identified  the  old  contract  number  for  items  on 
CREP  and  non-CREP  contracts  through  the  National  Item 
Identification  Number  (NUN).  The  CREP  metrics  worksheet  Warner 
Robins  ALC  supplied  to  HQ  AFMC  provided  CREP  contract  numbers. 
Non-CREP  contracts  were  those  repair  contracts  written  during  the 
CREP  initiative  that  Warner  Robins  ALC  did  not  classify  as  CREP. 
Only  those  NIINs  on  the  CREP  and  non-CREP  contracts  that  replaced 
a  recently  expired,  old  contract  became  part  of  the  analysis.  Changes 
in  prices  and  flow  days  for  CREP  and  non-CREP  items  were 
compared  between  the  old  and  new  contracts.  Some  NIINs  had 
more  than  one  just- expired  contract  written  within  the  same  year.  A 
NUN  could  have  several  CREP  contracts  if  the  first  contract  was 
written  or  modified  early  in  the  initiative. 

Warner  Robins  ALC  provided  copies  of  contracts  for  the  CREP  and 
non-CREP  contract  pairs  identified  from  G072D.2  The  analysis  con¬ 
sidered  only  items  that  had  nonzero  entries  for  items  repaired, 
scheduled  inputs,  and  repair  price.  All  price-related  data  were  con¬ 
verted  into  constant  dollars  (base  year  2000)  using  Secretary  of  the 
Air  Force  Financial  Management  price  deflators. 

In  all,  G072D  provided  data  on  375  unique  NIINs  from  182  contracts 
that  had  repairs  on  CREP  and  non-CREP  contracts  written  during  the 
CREP  initiative  and  their  respective  older  contracts.  Some  NIINs  ap¬ 
peared  on  several  CREP  contracts,  but  all  pairs  of  contract/ NIIN  data 
were  unique.  These  unique  pairs  of  contract  comparisons  produced 
319  observations  for  CREP  NIINs  and  135  observations  for  non-CREP 
NIINs.  This  sample  consisted  of  48  CREP  contracts  paired  with  60 
older  contracts  and  30  non-CREP  contracts  paired  with  44 
older  contracts.  The  data  set  included  48  of  the  74  CREP  contracts 


o 

The  contract  office,  headed  by  Jim  Grant,  provided  this  study  significant  assistance 
by  copying  297  repair  contracts.  We  extracted  data  from  those  contracts  and  contin¬ 
ually  referred  to  them  to  clear  up  questions  raised  by  data  ambiguities  in  other  infor¬ 
mation  systems. 
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reported  to  HQ  AFMC  or  65  percent  of  those  awarded  and  reported 
to  G072D  as  of  September  1998.3 

PRICE  AND  NEGOTIATED  FLOW  DAY  DIFFERENCES  FOR 
CREP  AND  NON-CREP  ITEMS 

The  results  of  these  comparisons  of  price  and  schedule  are  shown  in 
Figures  A.1  to  A.4.  Figure  A.l  compares  prices  for  NIINs  (n  =  319) 
from  48  CREP  contracts  with  their  just-expired,  old  contracts.  About 
66  percent  of  the  CREP  sample  of  NIINs  shows  a  reduction  in  repair 
prices.4  On  average,  prices  decreased  by  three  percent.  Half  of  the 
sample  had  repair  price  improvements  of  less  than  or  equal  to  18 
percent.  Over  8  percent  of  the  sample  showed  price  increases  over 
100  percent.  Changes  in  contract  design  may  account  for  most  of  the 
large  increases  in  this  comparison.  Some  contracts  went  from  firm- 
fixed  price  to  cost-plus-fixed-fee.  Others  had  CREP  contracts  with  a 
few  NIINs,  but  the  older  contract  had  many  NIINs. 

The  non-CREP  sample  of  135  NIINs  came  from  30  contracts.  Figure 
A.2  shows  that  62  percent  of  the  non-CREP  NIINs  had  repair  price 
improvements,  but  the  scale  of  those  improvements  differed  greatly 
from  CREP  NIINs.  On  average,  repair  prices  for  the  non-CREP  sam¬ 
ple  of  NIINs  increased  by  12  percent  compared  with  their  old  con¬ 
tracts.  Half  of  the  NIINs  in  our  sample  showed  repair  price  reduc¬ 
tions  of  four  percent  or  more,  with  very  few  NIINs  exhibiting  reduc¬ 
tions  greater  than  20  percent.  Over  30  percent  of  the  non-CREP 
NIINs  had  virtually  no  change  in  unit  repair  prices. 


o 

In  September  1998,  Warner  Robins  ALC  reported  that  101  contracts  had  incorporated 
one  or  more  CREP  tenets  during  the  CREP  initiative.  Of  those  101  contracts,  27  were 
written  by  the  U-2  Product  Directorate  and  primarily  classified.  We  included  U-2  con¬ 
tracts  in  the  behavioral  model,  because  the  CREP  tenets  incorporated  in  those  con¬ 
tracts  were  available  and  we  were  able  to  survey  the  appropriate  personnel.  The  data 
set  for  the  comparison  of  prices  and  schedule  for  items  before  and  after  the  CREP  ini¬ 
tiative  was  restricted  to  the  set  of  74  unclassified  contracts  written  by  “non-U-2”prod- 
uct  directorates  for  obvious  reasons  having  to  do  with  data  availability  and  reporting 
requirements.  Unclassified  repair  contracts  paid  for  by  the  revolving  fund  get  re¬ 
ported  to  G072D. 

4Repair  prices  excluded  government-furnished  materiel  (GFM)  costs. 
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Percentage  change  in  unit  repair  price 


Figure  A.1 — Sixty-Six  Percent  of  CREP  Sample  NIINs  Showed 
Reduced  Unit  Repair  Prices 
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Percentage  change  in  unit  repair  price 


Figure  A.2 — Sixty-Two  Percent  of  Non-CREP  NIINs  Showed  Improved 
Unit  Repair  Prices,  But  On  Average  Prices  Increased 
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The  next  two  figures  compare  flow  days.  Figure  A.3  shows  the  nego¬ 
tiated  flow  days  for  the  same  sample  of  CREP  NIINs  (n  =  319).  This 
sample  revealed  impressive  improvement — 58  percent  of  the  sample 
exhibited  improvements  in  negotiated  flow  days.  On  average,  CREP 
NIIN  negotiated  flow  days  decreased  by  14  percent  or  more.  Half  of 
the  CREP  sample  showed  flow  day  reductions  of  23  percent  or  more. 

Figure  A.4  indicates  that  60  percent  of  the  non-CREP  NIINs  in  our 
sample  had  reduced  negotiated  flow  days.  In  fact,  on  average,  ne¬ 
gotiated  flow  days  decreased  by  24  percent  for  our  non-CREP  sample 
of  NIINs.  Half  of  the  sample  showed  flow  day  reductions  of  20  per¬ 
cent  or  more. 


RMIDMR171 1-A.3 


Percentage  change  in  negotiated  flow  days 


Figure  A.3 — Fifty- Eight  Percent  of  CREP  Sample  NIINs  Also  Showed 
Reduced  Negotiated  Flow  Days 


STATISTICAL  ANALYSES  OF  PRICE  AND  NEGOTIATED 
FLOW  DAY  DIFFERENCES 

Are  these  results  statistically  meaningful?  To  examine  this  question, 
we  tested  whether  the  percentage  changes  for  CREP  and  non-CREP 
were  different  from  one  another.  We  also  tested  whether  CREP  and 
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Percentage  change  in  negotiated  flow  days 


Figure  A. 4 — Sixty  Percent  of  Non-CREP  Sample  NIINs  Showed  Improved 
Negotiated  Flow  Days 


non-CREP  price  and  flow  days  were  different  from  previous  contract 
values,  i.e.,  were  the  percentage  changes  different  from  zero?  We 
used  two-tailed  t-tests  to  evaluate  differences.  Table  A.l  shows  the 
CREP  and  non-CREP  repair  price  and  flow  day  values  used  in  our 
analyses. 


Table  A.l 

Repair  Price  and  Negotiated  Flow  Day  Statistics 
for  CREP  and  Non-CREP  Items 


CREP  items 
(N  =  319) 

Non-CREP 
items  (N  =  135) 

CREP  vs.  Non- 
CREP  Tests 

Repair  price  (%  change) 

Average 

-3.20 

11.81* 

* 

Standard  deviation 

(69.78) 

(62.35) 

Negotiated  flow  days  (%  change) 

Average 

-13.74*** 

-23.85*** 

* 

Standard  deviation 

(54.94) 

45.21 

NOTE:  Asterisks  indicate  significant  differences  from  previous  contracts:  *  denotes 
p  <  0.05;  **  denotes  p  <  0.01;  ***  denotes  p  <  0.001. 
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Prices 

A  t-test  evaluating  differences  between  CREP  and  non-CREP  per¬ 
centage  change  in  price  was  significant  (t  =  2.26,  p  <  0.02).  The 
means  indicate  that  CREP  price  changes  (average  percentage  change 
=  -3.20)  were  smaller  than  non-CREP  price  changes  (average 
percentage  change  =  11.81).  However,  CREP  repair  prices  were  not 
significantly  different  from  older  contracts,  i.e.,  they  were  essentially 
unchanged  (t  =  0.82,  p  >  0.05).  On  the  other  hand,  non-CREP  prices 
were  significantly  different  from  older  contracts  (t  =  2.23,  p  <  0.05). 
These  results  indicate  that  mean  CREP  repair  prices  were  not  signifi¬ 
cantly  different  from  mean  prices  for  the  same  items  on  previous, 
older  contracts.  Non-CREP  repair  prices  were  significantly  different, 
with  higher  mean  prices  for  items  on  non-CREP  contracts  compared 
with  the  same  items  on  older  contracts.  Thus,  while  repair  prices 
were  statistically  unchanged  for  CREP,  they  were  higher  with  non- 
CREP  contract  items. 

Next,  we  discuss  differences  in  schedule  or  negotiated  flow  days  for 
these  contracts,  comparing  old  and  new  contracts  for  the  same 
items. 

Negotiated  Flow  Days 

A  t-test  evaluating  the  difference  between  CREP  and  non-CREP  per¬ 
centage  change  in  flow  days  was  significant  (t  =  2.01,  p  <  0.05).  The 
means  indicate  that  the  percentage  decreases  in  non-CREP  flow  days 
(average  =  -23.85)  were  greater  than  percentage  decreases  in  CREP 
flow  days  (average  =  -13.74).  Both  CREP  and  non-CREP  items  had 
significant  decreases  in  flow  days  from  previous  contracts  (CREP:  t  = 
4.46,  p  <  0.001;  non-CREP:  t  =  6.13,  p  <  0.001).  These  results  indicate 
that  both  CREP  and  non-CREP  contracts  showed  improvements  in 
schedules  compared  with  their  previous  contracts. 

In  sum,  the  data  indicate  that  the  CREP  initiative  was  successful  at 
holding  prices  constant  over  time  while  reducing  flow  days.  Reduc¬ 
tion  of  flow  days  across  both  CREP  and  non-CREP  contracts  was 
strong,  but  price  increased  with  improved  schedule,  except  where 
CREP  was  implemented.  We  hypothesize  that  the  primary  effect  of 
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CREP  was  to  minimize  price  increases  while  reducing  negotiated 
flow  days,  an  explicit  goal  of  agile  logistics  and  the  primary  thrust  of 
the  CREP  initiative  at  Warner  Robins  ALC. 


Appendix  B 

INTERVIEW  QUESTIONS  FOR  CONTRACT 

REPAIR  TEAMS 


The  study  team  interviewed  individuals  who  had  participated  in 
Contract  Repair  Teams  (CRTs)  or  Integrated  Product  Teams  (IPTs) 
for  CREP  contracts.  Those  people  interviewed  were  identified  by 
Warner  Robins-ALC/PK  and  Warner  Robins-ALC/RE  as  innovative  in 
incorporating  CREP  tenets  in  repair  contracts.  Individuals,  who  were 
interviewed  in  groups,  were  sent  questions  before  the  meetings.  The 
questions  were  structured  but  somewhat  open-ended.  They  were 
meant  to  guide  discussions,  although  participants  were  encouraged 
to  volunteer  issues  the  researchers  had  not  anticipated.  The  inter¬ 
views  influenced  the  kinds  of  questions  our  survey  included  and 
their  wording. 

The  interview  questions  for  the  April  1998  and  fune  1998  interviews 
are  as  follows.  The  CRT  participants  were  asked  to  answer  the  ques¬ 
tions  in  light  of  a  specific  contract  the  ALC  considered  most  innova¬ 
tive  or  successful  in  incorporating  CREP  tenets. 

•  Contract  background 

•  Contract  number 

•  Contractor  name 

•  Number  of  contract  line  items  (CLINs) 

•  Contract  duration  (base  year  plus  number  of  options) 

•  Dollar  amount  (estimated) 

•  Contract  type 
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•  Repair  material  used  (government-furnished  material  [GFM]  or 
contractor-furnished  material  [CFM]). 

Initial  process  steps 

•  Requirements  determination/identification 

•  What  role  does  the  Program  Manager  play  in  this  process? 

•  Was  the  list  of  National  Stock  Numbers  (NSNs)  in  the  CREP 
contract  similar  to  the  previous  contract(s)? 

•  How  did  acquisition  reform/lean  logistics  or  other  CREP 
tenets  affect  this  part  of  the  process? 

•  Was  a  Statement  of  Objective  used  or  a  Statement  of  Work? 

•  Was  there  early  contractor  involvement? 

•  Purchase  Request  (PR)  package 

•  Primary  milestones  (date  started  and  completed) 

•  Funding 

•  What  funding  issues  were  involved  with  the  CRT’s  ability  to 
achieve  the  contract  design  goal  (the  lay- in  of  long-lead-time 
parts/rotable  pool,  etc.)? 

CRT/IPT 

•  Who  initiated  the  CRT  or  IPT  and  why? 

•  Who  were  the  core  members? 

•  How  often  did  the  team  meet? 

•  Was  there  an  advantage  to  using  a  CRT?  Disadvantages? 

•  Were  the  Defense  Contract  Audit  Agency  (DCAA)  and  the 
Defense  Contract  Management  Agency  (DCMC)  directly 
involved? 

•  How  did  the  contractor  participate? 

•  Did  all  members  of  the  CRT  work  toward  a  common  goal?  Were 
there  conflicts  between  CRT  and  functional  goals?  If  so,  how 
were  they  resolved? 


Interview  Questions  for  Contract  Repair  Teams  55 


Contract  design 

•  In  what  major  ways  is  this  contract  different  from  its  prede¬ 
cessor? 

•  What  innovations  were  incorporated  in  the  contract?  Improve¬ 
ments? 

•  What  other  innovations  were  contemplated  but  not  in¬ 
corporated? 

•  How  was  the  acquisition  strategy  determined?  Based  on  what 
goals? 

•  What  were  the  primary  challenges  in  contract  design?  How  were 
they  overcome? 

•  Was  a  cost  benefit  analysis  conducted?  If  so,  what  were  the 
results  and  how  did  they  influence  contract  design? 

•  Was  market  research  conducted?  If  so,  what  sources  were 
consulted  and  who  performed  the  analyses? 

•  What  major  “firsts”  were  accomplished  in  this  contract  or  during 
the  process? 

Lessons  learned 

•  What  did  the  CRT  learn  from  this  contract  experience? 

•  Did  the  CRT  consult  with  other  sources  to  learn  from  others 
before  writing  this  contract?  If  so,  what  sources  were  consulted? 

•  Have  the  lessons  learned  from  this  contract  influenced  the  way 
other  contracts  were  written?  If  so,  which  ones  and  how? 


Appendix  C 

CREP  SURVEY  CONDUCTED  AT  WARNER  ROBINS  ALC 


The  CREP  survey  conducted  at  Warner  Robins  ALC  in  1998  and  1999 
appears  in  the  next  six  pages.  Key  participants  in  the  contract  repair 
teams  (program  managers,  procurement  contracting  officers,  pro¬ 
duction  management  specialists,  and  item  management  specialists) 
or  those  individuals  who  had  most  influence  on  the  requirements 
and  contract  design  steps  of  the  CREP  process  filled  out  the  survey 
online  and  submitted  it  to  RAND  electronically.  The  survey  was  con¬ 
ducted  in  1998  and  again  in  1999  with  CRT  individuals  who  had  not 
completed  the  survey  the  first  time.  Questions  24  to  44  were  based 
on  issues  arising  from  acquisition  reform  and  agile  logistics  that  con¬ 
cerned  individual  behavior  and  factors  that  affected  behavior.1  Thus, 
they  applied  generally  to  all  of  the  CREP  contracts  in  which  the  indi¬ 
vidual  had  participated.  The  survey  questions  that  follow  provided 
the  data  for  the  organizational  levers  or  independent  variables.  The 
survey  is  shown  as  it  appeared  to  Warner  Robins  ALC  survey  partici¬ 
pants. 


Questions  1  to  23  (not  shown)  pertained  to  individual  contracts  and  the  contracting 
process.  These  responses  were  too  incomplete  to  use  in  the  study  and  were  dropped 
from  further  consideration. 
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Implementing  Acquisition  Reform  in  Depot  Maintenance  Contracts: 
Survey  of  Warner  Robins  ALC 


Your  Name:  I  I 

A.  Attitude  toward  Acquisition  Reform 

24.  How  well  do  you  understand  the  goals  of  acquisition  reform? 

Very  Somewhat  Not  Not  at  all 

well  Well  well  well  well 

o  o  o  o  o 

25.  How  supportive  are  you  of  the  goals  of  acquisition  reform? 

Very  Somewhat  Not  Not  at  all 

supportive  Supportive  supportive  supportive  supportive 

o  o  o  o  o 

26.  How  likely  is  acquisition  reform  to  contribute  to  the  following  outcomes  ( check  all  that  apply )? 


Very 

Somewhat 

Not 

Not  at 

Outcome 

likely 

Likely 

likely 

likely 

all  likely 

Improve  mission  capability  rates  for  end  user  O 

o 

o 

o 

o 

Reduce  the  cost  of  my  contracts 

o 

o 

o 

o 

o 

Increase  contractor  responsiveness/flexibility  O 

o 

o 

o 

o 

Increase  my  workload 

o 

o 

o 

o 

o 

Increase  my  job  satisfaction 

o 

o 

o 

o 

o 

Make  my  job  more  secure 

o 

o 

o 

o 

o 

B.  Leadership 

How  supportive  of  acquisition  reform  are  the  following  individuals  or  groups? 

Very 

Somewhat 

Not 

Not  at  all 

Individual/Group 

supportive  Supportive  supportive 

supportive 

Supportive 

Headquarters  AFMC 

o  o 

o 

o 

o 

Your  ALC  leadership 

o  o 

o 

o 

o 

Your  product  directorate  leadership 

o  o 

o 

o 

o 

Your  immediate  supervisor 

o  o 

o 

o 

o 
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28. 

How  true  are  the  following  statements  about  acquisition  reform? 

Very 

Somewhat  Not 

Not  at 

Statement 

true 

True 

true 

true 

all  true 

Air  Force  leadership  has  made  a  strong  case  for  why 
we  need  acquisition  reform. 

o 

o 

o 

o 

o 

Different  parts  of  the  chain  of  command  send 
different  messages  about  acquisition  reform 

o 

o 

o 

o 

o 

Management  attitudes  suggest  that  acquisition 
reform  is  the  latest  "program  of  the  month." 

o 

o 

o 

o 

o 

I  would  be  punished  for  a  failed  attempt  to  do 
something  innovative  with  a  contract. 

o 

o 

o 

o 

o 

I  would  be  rewarded  for  a  successful  attempt  to  do 
something  innovative  with  a  contract. 

o 

o 

o 

o 

o 

Hard  work  integrating  acquisition  reform  into  a 
contract  goes  unnoticed  by  management. 

o 

o 

o 

o 

o 

C.  Performance  Evaluation  and  Rewards 

29. 

How  important  are  the  following  factors  in  your  yearly  performance  evaluation? 

Very 

Somewhat 

Not 

Not  at  all 

Factor  important 

Important 

Important 

Important 

important 

Increasing  the  number  of  all  types  of  contracts 
processed  O 

o 

o 

o 

o 

Increasing  the  number  of  commercial  contracts 
processed  O 

o 

o 

o 

o 

Improving  the  timeliness  of  contracts  process 
(reduced  administrative  lead  time)  O 

o 

o 

o 

o 

Working  effectively  with  the  members  of  my 

CRT  O 

o 

o 

o 

o 

Improving  contract  performance  (cost, 
reliability,  cycle  times)  O 

o 

o 

o 

o 

Improving  my  educational  qualifications  O 

o 

o 

o 

o 

Improving  partnering  with  contractors  O 

o 

o 

o 

o 

30. 

How  likely  is  the  effective  use  of  acquisition  reform  to  lead  to  the  following  rewards? 
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31. 


32. 


Reward 

Very 

likely 

Likely 

Somewhat 

likely 

Not 

likely 

Not  at 
all  likely 

Higher  merit  increases 

o 

o 

o 

o 

o 

Improved  opportunities  for  promotion 

o 

o 

o 

o 

o 

Non-monetary  rewards  (e.g.,  time  off,  trips) 

o 

o 

o 

o 

o 

Recognition  from  ALC/product 
directorate/division  for  a  job  well  done 

o 

o 

o 

o 

o 

Official  acknowledgment  of  customer 
satisfaction 

o 

o 

0 

o 

o 

D.  Teaming  and  Partnering 

How  many  CRTs  (or  contract  IPTs)  have  you  participated  in  this  fiscal  year?l  I 

How  true  are  the  following  statements  about  the  CRTs  (or  IPTs)  you  have  worked  on? 


Not 

at 


Statement 

Very 

true 

True 

Somewhat 

true 

Not 

true 

all 

tnie 

The  CRT  was  effective  in  overcoming  functional  or 
disciplinary  barriers  to  cooperation. 

o 

o 

o 

o 

o 

Some  team  members  had  a  poor  understanding  of 
acquisition  reform. 

0 

o 

o 

o 

o 

Team  members  worked  toward  a  common 
performance  goal. 

o 

o 

o 

o 

o 

Non-core  CRT  members  (FM,  BC,  CR,  EN,  QA) 
were  well  integrated  into  the  process 

o 

o 

o 

o 

o 

33.  How  effective  and  responsive  were  the  following  individuals  or  groups  in  completing  their 
tasks  for  the  CRTs  (or  IPTs)  you  have  worked  on? 
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Acquisition  Reform  Survey 

Very 

Somew 

it  Not 

Not  at  all 

Individual/Group  effective 

Effective 

effectiv 

effective  effective 

Program  Manager/Logistics  Officer  O 

o 

o 

o 

o 

Item  Manager  O 

o 

o 

0 

o 

Production  Management  Specialist  O 

o 

o 

o 

o 

Procurement  Contracting  Officer  O 

o 

o 

o 

o 

Equipment  Specialist  O 

o 

o 

o 

o 

Engineering  O 

o 

o 

o 

o 

Quality  Assurance  O 

o 

o 

o 

o 

Financial  Management  O 

o 

o 

o 

o 

Contract  DMAG  O 

o 

o 

o 

o 

Competition  Advocate  O 

o 

o 

o 

o 

Small  Business  O 

o 

o 

o 

o 

Acquisition  Support  Team  (AST),  formerly 
known  as  the  RFP  Support  Office  (RFPSO)  O 

o 

o 

o 

o 

34.  How  true  are  the  following  statements  about  the  relationship  between  the  Air  Force  and 

contractors  for  the  contracts  you  have  worked  on? 

Very 

Somewhat 

Not 

Not  at 

Statement 

true 

True 

true 

true 

all  true 

A  positive  working  relationship  exists  with  the 
contractors 

o 

o 

o 

o 

o 

The  Air  Force  and  contractors  have  worked  together 
to  improve  processes  and  reduce  cycle  time. 

o 

o 

o 

o 

o 

Contractors  alert  the  Air  Force  to  anticipated 
problems  that  could  affect  the  contractor’s 
performance. 

o 

o 

o 

o 

o 

Contractors  show  a  willingness  to  assume  risks  in 
order  to  do  business  with  the  Air  Force. 

o 

o 

o 

0 

o 

The  Air  Force  shows  a  willingness  to  pay  for  the  risks 
it  wants  contractors  to  assume. 

o 

o 

o 

o 

o 

The  Air  Force  shows  a  commitment  to  reward 
contractors  who  perform  well. 

o 

o 

o 

o 

o 

E.  Training  and  Career  Development 

35.  ■  38.  How  many  years  of  work  experience  do  you  have  in 
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your  present  job?  I  1 

the  Air  Force?  □ 
other  non  Air  Force  government  positions?  I  1 
the  private  sector?  |  \ 

39.  -  40.  What  is  your  current  GS  grade  and  step? 

GS  grade:  1  I 
GS  step:  CZ3 

41.  About  how  many  days  of  formal  training  in  acquisition  reform  have  you  received  in  the  past 
two  fiscal  years? 

ONone 
O  1  to  2  weeks 
O  3  to  4  weeks 
OMore  than  one  month 

42.  How  often  do  you  receive  formal  or  informal  training  updates  on  acquisition  reform  as  it 
affects  your  job? 

O  Weekly 
O  Monthly 
O  Quarterly 
OBiannually 
O  Annually 

OLess  than  once  a  year 

43.  In  what  areas  is  a  lack  of  training  for  you  or  your  colleagues  slowing  the  implementation  of 
acquisition  reform  ( check  all  that  apply)? 

□  Requirements  determination 

□  Market  research 

□  Seeking  industry  input  into  RFP  development 

□  Determining  price  reasonableness 

□  Developing  the  PR  package  as  well  as  contract  terms  and  conditions 

□  Negotiation  of  contract  terms  and  conditions 

□  None  of  the  above 

44.  How  important  are  the  following  sources  of  information  for  keeping  current  changes  and 
"best  practices"  in  contracting? 
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Very 

Somewhat 

Not 

Not  at  all 

Factor 

important 

Important 

Important 

Important 

important 

DoD  Acquisition  Reform  Web  site 

o 

o 

o 

o 

o 

My  ALC  Web  site 

o 

o 

o 

o 

o 

Web  sites  at  other  ALCs  or  Headquarters 
AFMC 

o 

o 

o 

o 

o 

The  "home  office"  for  my  function  at  the  ALC 

o 

o 

o 

o 

o 

The  Acquisition  Support  Team  (aka  RFPSO)  at 
the  ALC 

o 

o 

o 

o 

o 

Other  personnel  in  my  product  directorate  or  at 
ALC 

o 

o 

o 

o 

o 

Contractors 

o 

o 

o 

o 

o 

Official  memos,  newsletters,  and  videos 

o 

o 

0 

o 

o 

Roadshows  or  Acquisition  Reform  Stand-down 
Week 

o 

o 

o 

o 

o 

|  Submit  |  If  you  have  any  questions  or  experience  problems, 
please  contact  Mary  Chenoweth  by  e-mail 
(mec@rand.org)  or  telephone  (310-393-0411, 
x6248). 


Appendix  D 

ORGANIZATIONAL  LEVER  VARIABLES 
FROM  SURVEY  DATA:  PRINCIPAL  COMPONENTS 

ANALYSES 


The  goal  of  the  principal  components  analyses  (PCAs)  was  to  identify 
the  key  organizational  lever  factors  that  could  be  described  by  a  lin¬ 
ear  combination  of  survey  questions.  PCAs  allow  one  to  examine 
relationships  among  responses  to  determine  whether  particular 
questions  reflect  the  same  underlying  concept.  These  analyses 
permit  data  from  numerous  questions  or  items  to  be  classified  into  a 
few  factors  that  describe  unique  scales.  PCAs,  rather  than  factor 
analyses,  are  an  appropriate  technique  to  use  when  the  analyses  are 
exploratory  in  nature  and  one  wants  to  account  for  total  variance  in¬ 
stead  of  shared  variance.  PCA  is  particularly  appropriate  here,  be¬ 
cause  we  had  no  prior  experience  with  how  people  would  respond  to 
or  interpret  the  survey  questions.  PCA  is  a  technique  commonly 
used  in  psychology  in  the  development  of  scales  of  personality  and 
intelligence  (Tabachnick  and  Fidell,  1996). 

Based  on  theoretical  assumptions  about  what  the  survey  was  trying 
to  measure,  data  were  analyzed  separately  for  questions  addressing 
different  organizational  levers.1  We  conducted  eight  different  PCAs 
to  identify  eight  lever  variables  and  then  computed  two  others  di¬ 
rectly  (i.e.,  not  using  PCA) — job  experience  and  a  product  directorate 
dummy  variable.  PCA  fundamentally  assumes  that  all  the  responses 


^CAs  conducted  on  all  29  question  subparts  created  factors  similar  to  those  based  on 
eight  separate  PCA  analyses.  These  results  indicate  that  the  survey  respondents 
perceived  our  theoretical  concepts  as  separate  or  distinct  factors. 
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are  measured  on  a  similar  continuous  scale  (e.g.,  five-point  scales). 
The  PCA  technique  identified  eight  organizational  lever  factors: 

•  Attitude  toward  acquisition  reform 

•  Leadership  consistency 

•  Performance  evaluations 

•  Performance  incentives 

•  Effective  teaming 

•  Contractor  partnering 

•  Air  Force  partnering 

•  Training  in  acquisition  reform. 

This  study  used  PCA  with  varimax  rotation  using  Kaiser’s  eigenvalue 
rule  (Nunnally,  1978),  which  states  that  only  factors  with  eigenvalues 
greater  than  one  are  retained.  This  rule  means  that  only  factors  that 
explain  more  variance  than  a  single  item  are  computed.  Eigenvalues 
express  how  much  variance  is  explained  by  each  factor;  principal 
components  compute  factors  so  that  the  first  factor  represents  the 
largest  amount  of  variance.  The  correlation  between  the  factor  and 
specific  items  is  reported  by  the  factor  loading  of  each  item.  The 
amount  by  which  each  variable  “loads”  on  a  factor  is  measured  by  its 
correlation  with  the  component.  Using  a  cutoff  value  of  0.60,  survey 
items  with  high  factor  loadings  were  retained  for  further  analyses, 
except  for  the  training  in  acquisition  reform  variable  where  the  two 
items  were  not  highly  correlated  (see  Table  D.l).2  The  regression 
analyses  retained  the  training  in  acquisition  reform  variable  even 
though  it  failed  to  meet  the  cutoff  criteria  because  of  its  importance 
in  implementing  new  business  practices. 

Varimax  rotation  maximizes  the  variance  of  the  squared  loadings  and 
is  the  most  common  orthogonal  rotation  method  (DeVellis,  1991). 


2Items  related  to  job  experience  and  training  did  not  load  on  the  same  factor.  Because 
we  were  interested  in  the  relationship  of  both  job  experience  (i.e.,  number  of  years  in 
present  job)  and  the  amount  of  training  Air  Force  personnel  had  received  with 
contract  innovation,  items  from  both  training  and  job  experience  were  retained  as 
distinct  organizational  levers,  and  thus  independent  variables  for  the  multiple 
regression  analyses. 
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Rotated  factor  patterns  and  Cronbach’s  coefficient  alphas 
(Cronbach,  1951)  were  computed  for  each  factor.  Cronbach’s  coef¬ 
ficient  alpha  is  a  widely  used  measure  of  reliability  in  which  alpha 
signifies  one  minus  the  error  variance.  Thus,  high  reliability  is  de¬ 
noted  by  alpha  values  close  to  one.  Table  D.l  shows  the  results  of 
the  PCAs  in  identifying  survey  questions  by  organizational  lever. 

Table  D.l 

Organization  Levers  Measured  by  Survey  Questions 


Organizational  Lever 

Factor 

Loading 

Alpha 

1.  Attitude  toward  acquisition  reform 

0.887 

How  supportive  are  you  of  the  goals  of  acquisition  reform? 

(Q25) 

0.694 

How  likely  is  acquisition  reform  to  contribute  to  the  following 
outcomes?  (Q26) 

•  Improve  mission  capability  rates  for  end  user 

0.833 

•  Reduce  the  cost  of  my  contracts 

0.807 

•  Increase  contractor  responsiveness/flexibility 

0.872 

•  Increase  my  job  satisfaction 

0.907 

•  Make  my  job  more  secure 

0.684 

2.  Leadership  consistency 

0.625 

How  true  are  the  following  statements  about  acquisition 
reform?  (Q28) 

•  Different  parts  of  the  chain  of  command  send  different 
messages  about  acquisition  reform 

0.853 

•  Management  attitudes  suggest  that  acquisition  reform  is 
the  latest  “program  of  the  month” 

0.853 

3.  Performance  evaluations 

0.887 

How  important  are  the  following  factors  in  your  yearly 
performance  evaluation?  (Q29) 

•  Improving  the  timeliness  of  contracts  processed  (reduced 
administrative  lead-time) 

0.843 

•  Working  effectively  with  the  members  of  my  CRT 

0.824 

•  Improving  contract  performance  (cost,  reliability,  cycle 
times) 

0.913 

•  Improving  partnering  with  contractors 

0.877 
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Table  D.l  (continued) 


Organizational  Lever 

Factor 

Loading 

Alpha 

4.  Performance  incentives 

0.901 

How  likely  is  the  effective  use  of  acquisition  reform  to  lead  to 
the  following  rewards?  (Q30) 

•  Higher  merit  increases 

0.873 

•  Improved  opportunities  for  promotion 

0.912 

•  Non-monetary  rewards  (e.g.,  time  off,  trips) 

0.863 

•  Recognition  from  ALC/product  directorate/division  of  a 
job  well  done 

0.776 

•  Official  acknowledgment  of  customer  satisfaction 

0.811 

5.  Effective  teaming 

0.678 

How  true  are  the  following  statements  about  the  CRTs  (or  IPTs) 
you  have  worked  on?  (Q32) 

•  The  CRT  was  effective  in  overcoming  functional  or 
disciplinary  barriers  to  cooperation. 

0.846 

•  Team  members  worked  toward  a  common  performance 
goal. 

0.801 

•  Non-core  CRT  members  (FM,  BC,  CR,  EN,  QA)  were  well 
integrated  into  the  process. 

0.689 

6.  Contractor  partnering 

0.873 

How  true  are  the  following  statements  about  the  relationship 
between  the  Air  Force  and  the  contractor?  (Q34) 

•  A  positive  working  relationship  exists  with  the  contractor. 

0.875 

•  The  Air  Force  and  contractor  have  worked  together  to 

0.859 

improve  processes  and  reduce  cycle  time. 

•  The  contractor  alerts  the  Air  Force  to  anticipated  problems 
that  could  affect  the  contractor’s  performance. 

0.864 

•  The  contractor  shows  a  willingness  to  assume  risks  in  order 
to  do  business  with  the  Air  Force. 

0.805 

7.  Air  Force  partnering 

0.718 

How  true  are  the  following  statements  about  the  relationship 
between  the  Air  Force  and  the  contractor?  (Q34) 

•  The  Air  Force  shows  a  willingness  to  pay  for  the  risks  it 
wants  the  contractor  to  assume. 

0.883 

•  The  Air  Force  shows  a  commitment  to  reward  contractors 
who  perform  well. 

0.883 
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Table  D.l  (continued) 


Organizational  Lever 

Factor 

Loading 

Alpha 

8.  Training  in  acquisition  reform 

0.392 

How  many  days  of  formal  training  in  acquisition  reform  have 
you  received  in  the  past  two  fiscal  years?  (Q41) 

0.789 

How  often  do  you  receive  formal  and  informal  training  updates 
on  acquisition  reform  as  it  affects  your  job?  (Q42) 

0.789 

9.  Job  experience 

n/a 

How  many  years  of  work  experience  do  you  have  in  your  current 
job?  (Q35) 

10.  U-2  Product  Directorate 

n/a 

Appendix  E 

STATISTICAL  RESULTS  OF  THE 
MULTIVARIATE  ANALYSES 


INTRODUCTION 

This  appendix  gives  the  results  of  an  ordinary  least  squares  (OLS) 
regression  with  the  four  CREP  innovation  groups  and  the  fifth  group 
that  includes  all  CREP  tenets  as  the  dependent  measures.  We  show 
two  sets  of  results  for  all  analyses.  The  first  half  of  Table  E.l  gives  the 
results  of  the  five  regression  models  using  101  observations,  which 
include  component  repair  and  U-2  sustainment  contracts.  The 
second  half  of  the  table  shows  the  results  for  the  sample  of  just  the 
component  repair  contracts  (without  U-2  sustainment  contracts,  n  = 
75).  Our  dependent  measure  is  the  number  of  innovation  group 
tenets  incorporated  in  the  contract  sample.  The  analyses  regressed 
the  dependent  variable,  i.e.,  the  number  of  tenets,  onto  the 
organizational  lever  variables.  Appendix  C  described  how  the  survey 
questions  were  used  to  construct  the  organizational  lever  variables. 

Because  our  dependent  variables  in  each  of  the  five  groups  were  not 
continuous,  but  rather  limited  in  range,  probit  analyses  were  also 
performed.  The  probit  results  were  strikingly  similar  to  the  results  of 
the  OLS  analyses  and  indicate  that  our  conclusions  are  robust  across 
both  estimation  methods.  The  OLS  analyses  offer  results  that  are 
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Table  E.l 

Results  from  the  OLS  Regression  Analyses:  Standardized  Coefficients  ((3)  by  Contract  Innovation  Group 


Contract  Innovation  Groups  (n  =  101;  component 

Contract  Innovation  Groups  (n  =  75;  component 

repair  and  sustainment  contracts) 

repair  contracts  only) 

Simple 

KeyAR 

Complete 

Agile 

All  CREP 

Simple 

KeyAR 

Complete 

Agile 

All  CREP 

Organizational  Levers 

Mods 

Concepts 

AR 

Logistics 

Tenets 

Mods 

Concepts 

AR 

Logistics 

Tenets 

Intercept 

0.28 

3.82 

7.28 

1.19 

7.38 

1.03 

3.97 

8.58 

2.18 

9.57 

Attitude  toward  ARa 

0.39 

0.31 

0.83 

0.52 

1.27 

0.75 

0.23 

1.41 

0.81 

2.20 

Leadership  consistency3 

0.20 

0.32 

0.32 

0.44 

0.64 

-0.01 

0.33 

0.07 

0.17 

0.11 

Performance  evaluation 

-0.11 

-0.24 

-0.21 

-0.65 

-0.74 

-0.26 

-0.33 

-0.34 

-1.21 

-1.32 

Performance  incentives 

-0.14 

0.09 

-0.19 

0.11 

-0.12 

-0.32 

0.32 

-0.81 

0.53 

-0.46 

Effective  teaming 

-0.27 

-0.77 

-0.91 

-0.75 

-1.51 

-0.40 

-0.93 

-0.96 

-1.12 

-1.93 

Contractor  partnering 

0.34 

0.11 

-0.12 

0.48 

0.47 

0.46 

0.22 

-0.26 

0.87 

0.68 

Air  Force  partnering 

-0.18 

0.07 

-0.04 

0.19 

0.08 

-0.26 

0.09 

0.07 

0.06 

0.05 

Training  in  AR 

0.42 

0.28 

0.53 

0.73 

1.20 

0.58 

0.30 

0.69 

0.92 

1.54 

Job  experience 

-0.04 

-.04 

-0.10 

-0.01 

-0.09 

-0.09 

-0.05 

-0.18 

-0.05 

-0.21 

Product  Directorate,  U-2 

0.02 

0.63 

1.47 

1.32 

2.47 

Adjusted  R2 

0.37 

0.20 

0.26 

0.55 

0.48 

0.50 

0.14 

0.15 

0.52 

0.39 

F-ratio 

6.76 

3.54 

4.45 

13.4 

10.12 

9.15 

2.37 

2.39 

9.66 

6.21 

Pr  >  F 

<.0001 

.0006 

<.0001 

<.0001 

<.0001 

<.0001 

.0222 

.0207 

<.0001 

<.0001 

p<0.01 


NOTE:  Coefficients  marked  white  with  a  black  background  have 
Coefficients  that  are  shaded  lightly  have  p  <  0.10. 

aVariable  reported  on  a  three-point  scale.  All  other  survey  variables  reported  on  a  five-point  scale. 


Coefficients  that  are  shaded  medium  have  p  <  0.05. 
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Statistical  Results  of  the  Multivariate  Analyses 
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easier  to  interpret  in  terms  of  variation  explained  by  R2  and  sr2; 
therefore,  only  they  are  presented  here. 1 

ORDINARY  LEAST  SQUARES  REGRESSION  RESULTS 

The  left-hand  column  of  Table  E.l  shows  the  independent  variable 
names  and  other  statistical  labels  for  the  analyses.  The  columns  to 
the  right  show  the  values  of  the  standardized  coefficients  or  (3s, 
which  indicate  the  marginal  effect  each  organizational  lever  has  with 
the  number  of  tenets  incorporated  in  CREP  contracts,  holding  all 
other  levers  constant.  For  example,  with  the  “simple  modification” 
model  (“simple  mods”),  on  average,  a  CRT  that  had  a  highly  positive 
attitude  toward  acquisition  reform  incorporated  0.39  more  tenets  in 
their  contracts  than  did  a  CRT  that  had  a  mid-range  attitude  (a  scale 
difference  of  one).  The  same  CRT  with  a  highly  positive  attitude 
toward  acquisition  reform  incorporated  0.78  more  tenets  than  did  a 
CRT  with  a  highly  negative  attitude  (a  scale  difference  of  two  or  two 
times  0.39). 2 

Near  the  bottom  of  Table  E.l  is  the  F-ratio,  which  measures  the 
amount  of  variance  explained  by  the  set  of  independent  variables. 
The  F-ratio  indicates  whether  the  independent  variables  explain  a 
significant  proportion  of  the  variance  in  the  dependent  variables.  In 
other  words,  it  shows  whether  the  organizational  levers  could  help 
explain  contract  innovation.  Indeed,  they  explained  all  four  sets  of 
contract  innovation  groups  along  with  the  group  of  all  CREP  tenets. 


^The  range  of  our  independent  variables  were:  simple  modifications,  0  to  3;  key  AR,  0 
to  6;  complete  AR,  0  to  9;  agile  logistics,  0  to  7;  all  CREP  tenets,  0  to  14.  Figure  2.2 
shows  the  histograms  of  these  dependent  variables.  Compared  to  ordinary  least 
squares  analyses,  the  probit  analyses  showed  improved  levels  of  significance  for  sev¬ 
eral  independent  variables — performance  evaluation,  effective  teaming,  and  contrac¬ 
tor  partnering. 

2We  note  that  two  variables — attitude  toward  AR  and  leadership  consistency — 
allowed  only  three  different  responses  after  mathematical  transformation,  which 
means  CRTs  can  be  at  the  most  two  scale  intensities  apart.  The  next  five  variables  in 
Table  E.l  allowed  five  different  responses,  which  means  CRTs  can  differ  by  as  much  as 
four  scale  intensities,  because  the  variable  allowed  five  different  responses. 
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Also  near  the  bottom  of  Table  E.l  is  the  coefficient  of  determination, 
R2,  adjusted  for  the  degrees  of  freedom.3  R2  measures  the  amount  of 
variance  in  the  dependent  variable  that  the  independent  variables 
explain.  Looking  across  the  four  sets  of  innovation  groups  and  the 
group  of  all  tenets,  the  agile  logistics  innovation  group  had  the  best 
fit  and  the  organizational  levers  explained  over  half  of  the  variance  in 
the  dependent  variable  (adjusted  R2  =  0.55).  This  innovation  group 
was  followed  by  the  groups  “all  CREP  tenets”  (adjusted  R2  =  0.48), 
simple  modifications  (adjusted  R2  =  0.37),  and  trailed  significantly  by 
the  two  acquisition  reform  groups  (adjusted  R2  =  0.20  and  0.26). 

These  results  tell  us  that  the  organizational  levers  available  to  senior 
leadership  were  more  related  to  agile  logistics  tenets  and  much  less 
related  to  acquisition  reform  tenets.  Two  possible  explanations  are 
suggested.  First,  the  organizational  lever  variables  associated  with 
acquisition  reform  may  have  been  omitted,  incomplete,  or  improp¬ 
erly  measured.  Second,  the  organizational  levers  the  survey  mea¬ 
sured  may  have  been  intrinsically  more  related  to  agile  logistics  than 
they  were  to  acquisition  reform.  We  know  through  interviews  that 
Warner  Robins  ALC  emphasized  two  types  of  contract  goals  during 
the  CREP  initiative:  (1)  simple  modifications  for  active  contracts  and 
(2)  agile  logistics  for  new  contracts.  Perhaps  combining  complicated 
concepts,  like  acquisition  reform,  with  simpler-to-explain  concepts, 
like  agile  logistics,  requires  a  more  complex  approach  to  applying  or¬ 
ganizational  levers. 

T-Test  Values 

The  t-test  on  the  organizational  lever  variables  in  the  regression 
analyses  explains  the  probability  that  the  coefficient  for  a  particular 
variable  is  not  equal  to  zero,  i.e.,  it  tells  us  whether  the  organizational 
lever  had  any  association  with  the  dependent  variable.  We  consid¬ 
ered  variables  as  having  explanatory  power  up  to  p  <  0.10.  We  chose 
this  cutoff  point  because  our  sample  of  101  observations  to  estimate 
coefficients  for  ten  independent  variables  is  relatively  small.  This 
cutoff  suggests  that  we  risk  being  wrong  10  percent  of  the  time  when 
p-values  reach  0.10. 


3The  R2  value  can  inflate  as  more  independent  variables  are  added,  even  if  they  are 
not  statistically  significant.  The  adjusted  R2  value  takes  into  account  the  addition  of 
variables  that  do  not  have  much  explanatory  power. 
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Squared  Semi-Partial  Correlations  (sr2) 

Table  E.2  shows  the  results  for  the  squared  semi-partial  correlations 
(sr2)  for  each  of  the  organizational  lever  variables.  Semi-partial  cor¬ 
relations  describe  the  amount  by  which  R2  is  reduced  if  a  particular 
independent  variable  is  deleted  from  the  regression  equation.  In 
short,  they  explain  the  unique  contribution  of  the  independent  vari¬ 
able  to  R2  (Tabachnick  and  Fidell,  1996).  The  higher  the  value,  the 
greater  contribution  the  variables  make  toward  explaining  variance 
in  the  dependent  variable.  Except  for  the  product  directorate 
dummy  variable,  training  in  acquisition  reform  is  the  organizational 
lever  variable  with  the  single  highest  explanatory  power.  Other  orga¬ 
nizational  lever  variables  associated  with  the  innovation  group 
variables  are  contractor  partnering  (for  the  simple  modifications 
model),  effective  teaming  (for  the  key  acquisition  reform  concepts 
and  agile  logistics  models),  and  performance  evaluation  (for  the  agile 
logistics  model). 


Table  E.2 

Results  from  the  OLS  Regression  Analyses:  Semi-Partial  Correlations  (sr2)  by  Contract  Innovation  Group 

Contract  Innovation  Groups  (n  =  101;  sustainment  Contract  Innovation  Groups  (n  =  75;  component 
and  component  repair  contracts)  repair  contracts  only) 


Organizational  Lever 

Simple 

Mods 

KeyAR 

Concepts 

Complete 

AR 

Agile 

Logistics 

All  CREP 
Tenets 

Simple 

Mods 

KeyAR 

Concepts 

Complete 

AR 

Agile 

Logistics 

All  CREP 
Tenets 

Attitude  toward  ARa 

0.035 

0.012 

0.028 

0.016 

0.029 

0.069 

0.005 

0.063 

0.025 

0.059 

Leadership  consistency3 

0.016 

0.022 

0.007 

0.020 

0.013 

0.000 

0.022 

0.000 

0.002 

0.000 

Performance  evaluation 

0.003 

0.009 

0.002 

0.029 

0.012 

0.010 

0.012 

0.004 

0.069 

0.026 

Performance  incentives 

0.003 

0.000 

0.000 

0.000 

0.000 

0.007 

0.005 

0.012 

0.006 

0.001 

Effective  teaming 

0.013 

0.057 

0.026 

0.025 

0.032 

0.014 

0.055 

0.022 

0.036 

0.033 

Contractor  partnering 

0.027 

0.001 

0.000 

0.013 

0.004 

0.028 

0.005 

0.002 

0.032 

0.006 

Air  Force  partnering 

0.008 

0.000 

0.000 

0.002 

0.000 

0.014 

0.001 

0.001 

0.000 

0.000 

Training  inAR 

0.121 

0.029 

0.034 

0.089 

0.075 

0.162 

0.030 

0.059 

0.129 

0.113 

lob  experience 

0.020 

0.013 

0.024 

0.000 

0.009 

0.072 

0.016 

0.069 

0.008 

0.038 

Product  Directorate,  U-2 

0.000 

0.049 

0.089 

0.103 

0.112 

aVariable  reported  on  a  3-point  scale.  All  other  survey  variables  reported  on  a  5-point  scale. 
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